The vizcrush Book

High-Performance Data Primitives for Browser Visualization

I wrote vizcrush because I got tired of watching browser tabs die.

If you’ve ever built a dashboard that loads 500K data points and watched Chrome’s “Page Unresponsive” dialog pop up: you know the feeling. The data is right there, the chart library is fine, but the browser just can’t handle it. That’s the problem this library solves.

This book is the story of how it solves it. Not an API reference (the library’s docs do that). This is the part you read front to back, once, and walk away understanding why the algorithms inside work the way they do, when to reach for each one, and what tradeoffs you’re making when you do.

Who this book is for

You will get the most out of this book if you’re:

• A frontend or full-stack engineer building dashboards or data-heavy UIs, and you’ve hit the wall where your chart library stops being fun.
• A data scientist or analyst who writes code and wants to understand what “downsampling,” “binning,” “sketches,” or “spatial indexing” actually mean in code instead of in papers.
• An engineer outside the web stack (backend, embedded, mobile) who’s curious about the specific tricks that make browser-side data viz fast.
• A student or self-learner working through “how do real libraries handle large datasets?”

You will probably bounce if you:

• Want a reference manual with every function signature documented. (The library repo has that.)
• Want a math-heavy treatment with proofs and formal error bounds. (A few papers are cited; go read those.)
• Want a “build a chart library” tutorial. This book intentionally stops at the data layer; the rendering is your business.

A note on prerequisites

This book is technical, but not gatekeeping. If a term is unfamiliar and this book doesn’t explain it, the best thing you can do in 2026 is ask your favorite language model (Claude, ChatGPT, whatever). They’ll give you a perfectly good definition of “SIMD,” “atomic operation,” “cache line,” or “Float64Array” in three seconds. This book deliberately doesn’t spend pages on those definitions; it spends them on the insights you can’t get from a definition.

That’s the deal: I’ll tell you the why: the thing a textbook won’t. You handle the what with whatever tool you prefer. Everyone’s time gets respected.

How to read it

• Chapters 1–3 set up the problem and the two engines (Rust/WebAssembly and the pure-JS core). Read these first, in order.
• Chapters 4–11 tour the algorithms, one family at a time. Each chapter is self-contained; skim the TL;DRs and dive into whichever ones are relevant to your problem.
• Chapters 12–13 are the “decide and build” chapters: pick the right tool, then see it in code. If you only read two chapters, read these two.
• Chapters 14–16 are the under-the-hood tour: fallbacks, performance lessons, and the math. Optional, but the fun part if you’ve already finished the rest.

Every chapter has a one-sentence TL;DR at the top. If the TL;DR doesn’t make you curious, skip the chapter. There’s no test at the end.

Let’s get into it.

This is the free, web-native edition of the book. The source lives at github.com/debug-diary-1/vizcrush-book. vizcrush itself lives at github.com/debug-diary-1/vizcrush.

The Wall

TL;DR: Chart libraries are great at 10K points and surprisingly bad at 100K. This chapter is about why that cliff exists, why the obvious fixes (workers, servers) don’t fix it, and where vizcrush fits in.

Here’s something nobody tells you when you start building data dashboards: chart libraries have a cliff.

D3, Plotly, ECharts, Chart.js: they all work beautifully at 10K points. At 50K they start to stutter on pan and zoom. At 100K you can see frames dropping. At 500K your users start filing bug reports. At a million points, the browser gives up entirely.

I call this the 100K Datapoint Wall.

The math behind it is simple and unforgiving. When a chart library renders a dataset, it does roughly this:

For every point in your data:
  1. Read the value                    → O(n)
  2. Convert to pixel coordinates      → O(n)
  3. Issue a draw call                 → O(n)
  4. Browser composites the result     → O(n)

Four passes, each touching every point. For a million points that’s four million operations, and you need all of them done in 16.6 milliseconds to hit 60fps. JavaScript on a single thread cannot do this. Not on an M3 Max, not on a Threadripper, not on anything.

And here’s the part that should bother you: most of that work is invisible.

A chart that’s 1920 pixels wide can show at most 1,920 distinct x-positions. If you have 2 million data points, roughly a thousand of them land on every single pixel. You’re computing coordinates, issuing draw calls, and burning CPU cycles for points that literally cannot be distinguished from each other on screen.

That’s not a rendering problem. That’s a data problem.

Why Workers Don’t Fix It

The knee-jerk reaction is “throw it in a Web Worker.” I tried this. Here’s what actually happens:

You call postMessage to send your data to the worker. JavaScript has to serialize that data, and serializing 100K objects takes about 150ms. The worker processes it (maybe 50ms), serializes the result back (another 50ms), and posts it to the main thread.

Total round trip: 250ms. For a single frame.

Now imagine the user is pinch-zooming on their phone, generating 10 touch events per second. Each one needs a fresh downsample. Your worker is perpetually behind, returning stale results for viewports the user has already scrolled past. It feels laggy no matter how fast the actual computation is because the serialization overhead dominates everything.

The problem isn’t compute speed. The problem is getting data across the boundary.

Why Server-Side Pre-Aggregation Has Limits

“Just aggregate on the server” sounds reasonable until you think about it:

• Round trip latency (50-200ms) is visible on every zoom gesture
• You need dedicated aggregation infrastructure that costs money
• Offline-first apps can’t phone home
• The server doesn’t know your exact screen width, pixel density, or current viewport bounds

Server aggregation makes sense for initial page load. It falls apart for interactive exploration.

The Actual Solution

vizcrush takes a different approach: do the computation in the browser but not in JavaScript.

The core algorithms are written in Rust and compiled to WebAssembly, with a pure-JavaScript implementation of every algorithm alongside it — a “fallback” that, in some browsers, turns out to be the faster path (chapter 3 has the surprising numbers). A thin JavaScript layer detects what the environment supports and picks accordingly.

No serialization. No network calls. No Web Worker boundary. The WASM module reads directly from the same typed array memory that JavaScript uses. A million-point LTTB downsample takes about 2 milliseconds.

That’s the 30-second pitch. The rest of the book is how.

What vizcrush Actually Does

TL;DR: vizcrush is a data-compute layer, not a chart library. Numbers in, numbers out. You keep your chart library, whichever one it is.

I want to be very precise about this because it trips people up: vizcrush does not render anything.

No canvas. No SVG. No DOM nodes. No WebGL draw calls. Nothing visual.

It takes a big array of numbers in, and returns a smaller (or differently shaped) array of numbers out. What you do with those numbers is your business: feed them to D3, Plotly, Chart.js, ChartGPU, a plain <canvas>, whatever. vizcrush doesn’t care and doesn’t want to know.

Think of it as a food processor. You put raw ingredients in. It chops, slices, and dices. It doesn’t cook anything or plate it for you. That’s the chart library’s job.

Here’s the layer cake:

┌─────────────────────────────────────────────┐
│  Rendering: ChartGPU, D3, Plotly, ECharts   │  ← draws pixels
├─────────────────────────────────────────────┤
│  Data Compute: vizcrush                     │  ← THIS LAYER
├─────────────────────────────────────────────┤
│  Browser Compute: WebAssembly, JS engines   │  ← hardware
└─────────────────────────────────────────────┘

The rendering layer is mature. The browser compute layer is mature. The data compute layer in between was missing. That’s the gap.

The Zero-Loss Thing

People hear “downsampling” and think “lossy compression.” It’s not. Let me be clear about what happens:

Your original 2-million-point dataset stays in memory, untouched. vizcrush creates a view, a selection of points optimized for the current screen. When the user zooms into a small region, vizcrush re-selects from the full dataset for that region. Every zoom level gets pixel-perfect representation.

A human looking at the downsampled chart cannot distinguish it from the full-resolution chart. That’s not a marketing claim, it’s a mathematical property of the LTTB algorithm. The points are selected specifically to maximize visual fidelity for the given number of output pixels.

Your data is safe. We just pick the points that matter.

Now, how do we pick them quickly? There are two very different machines we can point at the problem, and the next chapter is about when to use which.

The Two Engines: WASM and the JS Core

TL;DR: Every vizcrush algorithm ships twice: once in Rust compiled to WebAssembly, once in plain JavaScript over typed arrays. In Chrome, WASM is about 4× faster. In Firefox and Safari, the JS core holds its own or outright wins. vizcrush picks per call; you never have to think about it.

Conventional wisdom says WebAssembly runs at “near-native speed” and JavaScript is the slow interpreted cousin. We believed it too. An early draft of this chapter told a much tidier story, complete with a crossover table showing exactly when each engine wins. Then we actually benchmarked across three browser engines, and the tidy story fell apart. What replaced it is more interesting, and it’s the real content of this chapter: performance claims you haven’t measured are fiction, even when they’re plausible.

Engine 1: Rust compiled to WebAssembly

WebAssembly is a binary instruction format that browsers execute without the overheads JavaScript carries: no garbage collector pausing your code at random moments, no JIT compiler deciding halfway through that your function needs recompiling because you passed it a string instead of a number.

We write the core algorithms in Rust and compile them to .wasm modules (8–27KB gzipped per package). The build enables the SIMD128 feature, and here’s the first honest surprise: enabling it changes nothing. The SIMD build is byte-identical to the scalar build. SIMD (Single Instruction, Multiple Data) is real and powerful, but compilers only auto-vectorize loops with no data-dependent branching, and downsampling loops branch constantly (“which of these candidate points makes the biggest triangle?”). You don’t get SIMD “for free” by flipping a compiler flag; you get it by restructuring algorithms around it, which is future work, not shipped reality.

Engine 2: The JS core everyone underestimates

The JavaScript implementation is not a grudging fallback. It’s the same algorithm, written over Float64Array with monomorphic inner loops, which happens to be exactly the shape of code modern JIT compilers optimize best. No boxed objects, no property lookups, no polymorphism: just flat memory and arithmetic. After warmup, a JIT turns that into machine code that looks a lot like what the Rust compiler produced.

What We Measured

Batch-timed LTTB, one million points down to one thousand, three engines:

Read that middle row again. In Firefox, the “fast” engine is eight times slower than the “fallback.” And in every engine, WASM’s first call is the slowest, because module instantiation and warmup aren’t free.

The lesson: “WASM vs JS” is not a property of your code. It’s a property of the engine running it. Any library that quotes you a single universal speedup number for WASM is telling you about one browser.

So Why Keep WASM at All?

Because Chromium is where most users are, and a 4× win in the common case is worth shipping. The decision rule is simple: use WASM when WebAssembly is available, keep the JS core first-class and parity-tested (same inputs, same outputs, verified in CI), and let either one carry the load without the caller noticing. Below a small data size the kernel prefers the JS core anyway, since crossing the WASM boundary has a fixed cost that tiny inputs never pay back.

What About the GPU?

You may have noticed a missing engine. Binning half a million points is embarrassingly parallel; GPUs were built for exactly that; WebGPU exposes them to the browser. There are literally WGSL compute-shader drafts sitting in the vizcrush repository.

For bin2d — the most GPU-favourable of the five — we eventually wired the draft up to settle the question with data instead of arithmetic. It ships as an opt-in (backend: "webgpu", silent fallback when the GPU can’t run), and the measurement was unambiguous: correct to within a few f32 edge effects, and roughly 15× slower than WASM even taking its best run, at 100K, 1M, and 5M points alike. GPU compute pays a fixed toll — upload the data to GPU memory, dispatch the shader, read the results back — and every operation in this book already finishes in single-digit milliseconds on the CPU, so there is nothing for the GPU’s parallelism to amortize. The CPU is done before the GPU finishes clearing its throat. (The numbers live in the vizcrush repo as ADR 0004.)

The economics flip only when the data already lives on the GPU — a rendering pipeline consuming the grid without ever reading it back — or when several GPU steps chain against a single upload. The other four drafts wait for that day.

So it’s not “GPU good, CPU bad,” and it’s not “WASM fast, JS slow.” It’s: measure, on the engines your users actually run.

Speaking of measuring, the first algorithm we’re going to meet is a beautifully sequential one, which means the question of parallelism never even comes up. It’s also the one that got me into writing this library in the first place.

LTTB: The Algorithm That Started Everything

TL;DR: LTTB picks the points that matter most to the human eye by thinking in triangles. It’s simple, it’s fast, and it’s the downsampling algorithm you should reach for by default.

Imagine a chart showing a million temperature readings across a year. Your screen is 1920 pixels wide. You have to throw away all but 1920 points. Which ones do you keep?

The obvious answer, “every five-hundredth one”, is also the wrong one. Uniform sampling will cheerfully land on flat, boring regions and step right past the year’s hottest day and coldest night. You just hid the two most important moments of the year from your user.

A guy named Sveinn Steinarsson wrote a master’s thesis in 2013 about a smarter way to pick. He called it Largest-Triangle-Three-Buckets, LTTB for short. It’s the algorithm vizcrush reaches for by default, and the one I reach for maybe 95% of the time I’m downsampling anything. It’s also beautifully simple once you see the trick.

The trick is: think in triangles.

The Triangle Trick

LTTB chops your time series into roughly equal buckets, one bucket per output point, minus the first and last, which are always kept. From each bucket it picks one survivor: the point that forms the biggest triangle with its neighbors.

What triangle? Three vertices:

1. Left vertex: the point we selected from the previous bucket
2. Right vertex: the average of all points in the next bucket
3. Candidate vertex: each point in the current bucket, tried one at a time

        candidate point
            *
           /|\
          / | \
         /  |  \        ← area of this triangle
        /   |   \
       /    |    \
      *─────┼─────*
  previously       average of
   selected        next bucket

The candidate point that creates the biggest triangle is the one that deviates most from the straight line connecting its neighbors. In other words, it’s the most “interesting” point, the peak of a spike, the bottom of a dip, the sharpest bend in a curve.

This is why LTTB works so well. It’s not randomly sampling. It’s not blindly decimating. It’s specifically selecting the points that contribute the most visual information.

The Algorithm, Step by Step

1. Always keep the first point.

2. For bucket i (from 0 to threshold-2):
   a. Look at the NEXT bucket (i+1) and compute its average (x, y).
   b. For each point in the CURRENT bucket:
      - Compute the triangle area between:
        (previous selected point, current point, next bucket average)
   c. Keep the point with the largest area.
   d. That point becomes the "previous selected" for the next iteration.

3. Always keep the last point.

The entire thing is a single pass through the data. O(n) time. No sorting needed (the data is already time-ordered). No extra memory beyond the output buffer. It’s gorgeous.

Why the Next-Bucket Average?

This is the clever bit. By looking ahead to the average of the next bucket, LTTB accounts for what’s coming next. A point that’s interesting in isolation might be redundant given what follows. The look-ahead prevents selecting a point that the next bucket will make obsolete.

This forward-looking property is also why LTTB can’t be parallelized on the GPU. Each bucket depends on the next bucket’s average, which creates a dependency chain.

The Triangle Area Formula

The triangle area can be computed without square roots or trigonometry using the cross-product form:

area = |(prev_x − next_x) · (curr_y − prev_y)
       − (prev_x − curr_x) · (next_y − prev_y)|

Three multiplications, five additions, one absolute value per candidate point. That’s why LTTB is fast: the inner loop does almost no work.

The same trick shows up later in the book under a different name (the math chapter at the end derives it properly). For now, just know that this is what makes a million-point downsample take under two milliseconds.

LTTB is the right default for smooth data. But trading charts have spikes. Sensors have glitches. Crypto prices do things that LTTB will happily smooth over. The next chapter is about what to do when the “average” behavior isn’t what the user needs to see.

MinMaxLTTB, M4, LTOB: When LTTB Isn’t Enough

TL;DR: LTTB smooths spikes. For data where the spikes are the story, you need a different algorithm. This chapter covers the three understudies and when to cast each one.

The first time I shipped LTTB to a real user, I got a Slack message within an hour: “your chart is lying to me.”

The user was watching BTC prices. In their raw data, there was a moment where the price stabbed up $2,000 in a single candle and came back down just as fast. In my beautifully downsampled chart, that spike was gone, smoothed into a gentle curve that suggested nothing interesting had happened.

LTTB didn’t have a bug. It did exactly what it was designed to do: pick the points that look most “interesting” relative to their neighbors. The problem is that a single extreme value, surrounded by normal ones, doesn’t look very interesting to an algorithm measuring triangle areas. The trend matters more than the moment. So the moment disappears.

The user didn’t care about the math. They cared that a trading decision had just been made on a lie. I spent that weekend reading papers.

MinMaxLTTB: Don’t Drop the Spikes

I first hit this problem with crypto trading data. BTC price can spike $2,000 in a single candle and recover just as fast. LTTB’s triangle-area metric tends to smooth over these spikes because the overall trend matters more to the triangle than a momentary extreme.

MinMaxLTTB fixes this with a two-phase approach:

Phase 1, MinMax pre-selection. Divide the data into 4× the target number of buckets. From each bucket, unconditionally keep the point with the minimum y and the point with the maximum y. This guarantees every spike is in the candidate set, no matter how brief.

Phase 2, Run LTTB on the pre-selected points. Now LTTB does its visual-shape optimization on a dataset where the spikes are already locked in.

The result: you get LTTB’s visual quality plus guaranteed spike preservation. The cost is that Phase 1 adds a pass through the data, but it’s a simple min/max scan, cheap.

Use MinMaxLTTB for financial data, IoT sensor alarms, or anything where missing a momentary extreme would be a bug.

M4: Four Points Per Pixel

M4 takes a totally different philosophy. For each output bucket, it keeps exactly four points:

1. The first point (left temporal boundary)
2. The last point (right temporal boundary)
3. The minimum y (the low)
4. The maximum y (the high)

If you squint, this is a candlestick chart, open, close, low, high, for each pixel column. It’s guaranteed to preserve exact extremes. The downside is it produces up to 4× more output points than LTTB for the same number of buckets.

When to use it: when your compliance team says “the chart must show the exact minimum and maximum values that occurred” and they don’t care about visual elegance.

LTOB: LTTB’s Lazy Cousin

LTOB (Largest-Triangle-One-Bucket) simplifies LTTB by using the current bucket’s own boundary points as the triangle vertices instead of looking ahead to the next bucket’s average.

It’s about 20% faster and 5-10% worse visually. I honestly don’t recommend it over LTTB unless you’re on extremely constrained hardware and every microsecond counts. But it’s there if you need it.

Picking the Right One

When someone asks me “which algorithm should I use,” I ask one question: does the data have spikes that matter?

• No spikes, smooth trends → LTTB
• Spikes matter (finance, IoT alarms) → MinMaxLTTB
• Must preserve exact min/max (compliance) → M4
• Maximum speed, quality secondary → LTOB

vizcrush has an auto_downsample function that makes this decision based on a hint you provide ('time_series', 'financial', 'sensor', 'scatter').

All four of these algorithms assume your data has a natural x-axis to thin out along, time, distance, index, something. Scatter plots don’t have that luxury. You can’t “downsample along X” when X and Y are independent. That’s a whole different problem, and it has its own chapter next.

Binning: When Scatter Plots Lie To You

TL;DR: A scatter plot of a million points looks the same as ten. Binning turns the overlap problem into a density heatmap, and reveals the clusters that were hiding all along.

Pour a glass of water into a second glass that already has water in it. Now tell me how much water is in the second glass.

You can’t, really. Once the water mixes, there’s no “this is the new water, this is the old water.” Scatter plots have the same problem: once two dots land on the same pixel, the pixel has no idea whether it’s holding two points or two thousand. The canvas loses information the moment the pen touches it.

The consequence is surprisingly ugly. A scatter plot with 500K points can look identical to one with 5K points. The middle fills up with ink, and visual density goes to maximum everywhere there’s data. Real clusters are invisible. Real gaps are invisible. Your chart is a Rorschach test.

Binning is the fix. Instead of plotting dots, we count them, and then we plot the counts.

1D: The Humble Histogram

The simplest bin: divide a range into N equal segments, count how many values fall in each.

Range [0, 100], 10 bins:
  Bin [0-10):  45 values   ████████████████
  Bin [10-20): 12 values   ████
  Bin [20-30):  8 values   ███
  ...

Just division and counting. O(n), single pass.

The only tricky part is edge cases: what happens when a value equals the maximum? It would compute bin_index = num_bins, which is out of bounds. We clamp it to num_bins - 1. This puts the max value in the last bin, which is the least surprising behavior.

2D: Density Grids

bin2d extends this to two dimensions. The output is a flat grid:

grid[row * num_cols + col] = count of points in that cell

For a 128×128 grid, you get 16,384 cells. Each cell holds a count. You can color-map these counts to produce a heatmap. Suddenly those invisible overlapping dots reveal clusters, corridors, and density gradients.

Binning is also as parallel as problems get: each input point’s bin assignment is independent of every other point’s. That’s why GPU people love it — a compute shader can dispatch one thread per point:

let xi = u32((px - x_min) / x_range * f32(x_bins));
let yi = u32((py - y_min) / y_range * f32(y_bins));
atomicAdd(&grid[yi * x_bins + xi], 1u);

The atomicAdd is essential in that world. Multiple GPU threads might try to increment the same cell simultaneously; without atomic operations you’d get race conditions and lost counts. Worth knowing — and vizcrush ships exactly this shader, wired, as an opt-in (backend: "webgpu"). Measuring it is what proved the point: end-to-end it runs ~15× slower than WASM at every realistic size, because upload and readback dominate (chapter 3 tells the story; ADR 0004 in the vizcrush repo has the receipts). The default bin2d runs on WASM/JS.

Hexagonal Binning

If you put a rectangular grid on a scatter plot, you get visual artifacts at the corners. Diagonal neighbors of a square are √2 times farther than edge neighbors, which creates directional bias: clusters look square instead of round.

Hexagons fix this. Every neighbor of a hex is equidistant. The grid tiles the plane more uniformly, and the resulting heatmap looks more natural.

The coordinate math is a bit more involved (you need to account for alternating row offsets), but the binning logic is the same: compute which hex center is closest, increment its count.

Binning turns a wall of overlapping dots into a legible density field. But sometimes you don’t want a summary, you want to interrogate specific points. “Which of these points is under my cursor?” “Give me everything in this viewport.” That’s the job of the next chapter.

Quadtrees, kd-trees, Hash Grids: Finding Needles Fast

TL;DR: When the user hovers a mouse over a million points, you can’t check all million in time. Spatial indexes organize points so most of them get skipped without ever being examined.

Think about how you’d find a book in a physical library. You don’t scan every spine on every shelf, that would take days. You walk to the section labeled “Fiction,” then “F,” then “Fitzgerald.” Each step cuts the search space by a huge factor. By the time you’ve taken three steps, you’re looking at a dozen books instead of a hundred thousand.

Spatial indexes are libraries for points. Before you query anything, you spend some time organizing the data into a hierarchy, each level narrower than the last. When a query comes in, “what’s under the cursor?”, you walk that hierarchy the same way you walked the library. Most of the data gets eliminated before you look at it.

This chapter covers four ways to build that library: quadtrees, kd-trees, spatial hash grids, and Morton codes. They’re all solving the same problem with different tradeoffs.

Quadtree: Divide and Conquer in 2D

A quadtree recursively splits 2D space into four quadrants. You keep splitting until each leaf holds a manageable number of points (we use 64 as the threshold).

Level 0:  One node covering the entire space
Level 1:  Four quadrants (NW, NE, SW, SE)
Level 2:  Sixteen sub-quadrants
Level 3:  Sixty-four regions
...
Level 12: Up to 16 million regions (max depth)

Range query (find all points in a viewport):

Start at the root. If the viewport doesn’t intersect this node’s bounds, skip the entire subtree. You just eliminated a quarter of the data in one comparison. If the viewport fully contains the node’s bounds, return all points in the subtree without checking individually. Otherwise, recurse into children and check individual points.

For a viewport showing 1% of the total area, this typically examines about 3-5% of the tree nodes, skipping the other 95%. On a million points, instead of checking all million, you check maybe 50K. That’s the difference between 1ms and 50 microseconds.

Building the tree costs O(n log n) but you only do it once. Queries are O(log n + k) where k is the number of results.

kd-tree: Better for Nearest Neighbors

A kd-tree splits space along alternating axes. Level 0 splits on X at the median. Level 1 splits on Y. Level 2 splits on X again.

The key advantage over quadtrees: kd-trees partition at the median, creating balanced trees. Quadtrees partition at the spatial center, which creates unbalanced trees when points cluster.

For k-nearest-neighbor queries (hover tooltips), kd-trees are faster because the binary structure allows tighter pruning. (vizcrush today serves kNN from the quadtree — a Rust kd-tree draft exists in the repo, unwired; the pruning idea below is why it might get wired someday.) You search the closer child first, establish a candidate distance, then only search the farther child if the splitting plane is closer than your current best, which it usually isn’t.

Morton Codes: GPU-Friendly Spatial Ordering

The Core Idea, Told With a Road Trip

Suppose you want to visit every house in a city in some reasonable order, not randomly bouncing around, but walking neighboring houses in sequence. A good order would be: finish this block, finish the next block, finish the quadrant, move to the next quadrant. You want a path where consecutive stops are physically close to each other.

That path exists, and it looks like a jagged “Z” repeated at every scale, inside each quadrant, inside each sub-quadrant, all the way down. Mathematicians call it the Z-order curve. Drive along it and you visit every location exactly once, with each consecutive pair close to each other.

Morton codes are the addresses along that Z-shaped path. If you assign each 2D point a Morton code, then sort all your points by Morton code, your data magically arranges itself so nearby points sit next to each other in the array. You just turned a 2D spatial problem into a 1D sorting problem, and 1D sorting is something GPUs are phenomenally good at.

How You Compute One

The trick for turning an (x, y) pair into a single number is surprisingly simple: interleave the bits of x and y. Take x’s bits and shift them so they sit in the even positions of the output. Take y’s bits and shift them into the odd positions. Merge.

x = 5  →  binary 101
y = 3  →  binary 011

Interleaved: (x bits in even positions, y bits in odd positions)
             1 0 1 0 1 1 → 101011
             ↑   ↑   ↑
             x   x   x

The resulting number has the beautiful property that points which are close in 2D are mostly close in Morton order (the Z-curve has a few long jumps at quadrant boundaries, but they’re rare). Close enough, in any case, that sorting by Morton code gets you a tree-building-friendly ordering for free.

Why This Shape Matters

Building a traditional quadtree is inherently recursive, and recursion is hostile to every kind of parallel hardware: parallel workers want to do the same work on different data. Morton codes turn tree construction into two operations that parallelize beautifully: compute a code for each point (no coordination between points), then sort the results.

vizcrush computes Morton codes in a single flat pass over the input in the WASM/JS compute layer — the textbook definition of “embarrassingly parallel,” even when executed sequentially. That structure is the point: it makes the algorithm cache-friendly today, and it’s the shape you’d hand to a GPU if profiling ever justified one (it hasn’t yet — chapter 3).

Spatial Hash Grid: When You Don’t Want a Tree

Trees are great, but they have two costs that hurt for certain workloads. They take O(n log n) to build, and every query walks a hierarchy of nodes. If your data is roughly uniform and your query radius is roughly the same size every time, think “find all neighbors within 5 pixels of the cursor” or “which points are inside this brush stroke”, you can do better with no tree at all.

A spatial hash grid divides 2D space into fixed-size cells. Each cell holds a list of point indices. There’s no hierarchy, no recursion, no balancing. Insertion is O(1) per point:

cx = floor(x / cell_size)
cy = floor(y / cell_size)
key = hash(cx, cy)
cells[key].append(point_index)

The cell hash combines cx and cy into a single 64-bit key using two large primes. This is the standard trick from Teschner’s “Optimized Spatial Hashing”:

hash(cx, cy) = (cx · 73856093) XOR (cy · 19349663)

The primes are chosen so that nearby cells get very different hash values, distributing them evenly across the underlying hash table. No clustering, no resize storms.

Radius queries are stupidly direct: compute which cells the query circle overlaps, iterate over their indices, and reject points that are inside the bounding box but outside the actual radius. For a query radius matched to the cell size, you visit at most 9 cells (3×3) regardless of how many points you’ve inserted. A million-point dataset answers a “neighbors within r” query in tens of microseconds.

The catch: hash grids fall over when the radius is much larger or smaller than the cell size. Query radius >> cell size means you visit thousands of cells. Query radius << cell size means each cell holds way more candidates than you actually need. The cell size has to be tuned to the query pattern. If your queries are heterogeneous (sometimes brushing 5 pixels, sometimes selecting a viewport), use a kd-tree instead. If they’re homogeneous, the hash grid will smoke any tree.

Picking the Right Spatial Index

All four of these structures assume your data just is, a fixed set of points sitting still. But in the real world, new data arrives every second and old data ages out. That’s a different world, and it has its own set of algorithms. Turn the page.

Streaming and Sketches: Numbers That Never Stop

TL;DR: When data arrives forever, you can’t keep it all. Streaming algorithms and sketches let you answer hard questions, percentiles, distinct counts, frequencies, with a fixed amount of memory no matter how long the stream runs.

Imagine trying to count every car that drives past your house for the rest of your life. Not a bad challenge. Add a twist: keep a running average of how fast they were going, a running list of the fastest and slowest, and an estimate of the 99th-percentile speed, all without writing anything down except a single Post-it note.

It sounds impossible. It’s not. The algorithms in this chapter do something like that every day, for terabytes of live data, on fixed memory budgets. They’re small, elegant, and deeply clever. Most of them are 60-year-old math that got rediscovered when streaming data became normal.

Real-time dashboards are a different beast from the batch work in the rest of the book. Data arrives continuously: a server monitoring system might push 1,000 metrics per second. You need rolling statistics (min, max, mean, standard deviation) that update instantly, plus a chart that stays smooth at 60fps despite constant data churn.

The naive approach recalculates from scratch on every batch: scan all 20K buffered values for min/max, compute the mean, compute the variance. That’s O(n) per update. At 20 updates per second on a 20K buffer, that’s 400K operations per second wasted on re-computation.

Welford’s Algorithm: One Pass, No Surprises

Carl Welford published this in 1962 and it’s still the best way to compute running variance. Here’s the idea:

When a new value arrives, you can update the mean and variance without re-scanning the entire dataset. Each update is O(1):

New value x arrives (count becomes n):

  delta  = x - old_mean
  mean   = old_mean + delta / n
  delta2 = x - mean                ← note: uses the NEW mean
  M₂     = M₂ + delta * delta2

  variance = M₂ / (n - 1)
  std_dev  = sqrt(variance)

The brilliant part is using both delta (difference from old mean) and delta2 (difference from new mean). Their product is always positive, which means M₂ always increases. No catastrophic cancellation, no accumulated rounding errors. I’ve seen implementations using the textbook formula variance = E[x²] - E[x]² produce negative variance on large datasets because of floating-point cancellation. Welford’s never does that.

The Rolling Window

vizcrush’s streaming stats maintain a circular buffer. When the buffer is full, adding a new value evicts the oldest one. We need to remove the old value’s contribution from the running statistics.

Removing from Welford’s is the reverse operation: subtract the old value’s delta contribution from M₂, then update the mean. It works, but there’s one gotcha: if the evicted value was the current min or max, we need to rescan the buffer to find the new extreme. This rescan is O(window_size), but it’s infrequent (only when the min or max ages out).

appendAndDownsample: The Streaming Sweet Spot

For streaming charts, the common pattern is:

1. New batch arrives (50 points)
2. Append to rolling buffer
3. If buffer exceeds max size, drop oldest
4. Downsample entire buffer to display-point count
5. Update chart

vizcrush combines steps 2-4 into a single function. No temporary buffer allocations, no double iteration. The LTTB pass operates directly on the combined data.

t-digest: Percentiles Without Sorting

If someone asks “what’s the 99th percentile of the last million values,” you can’t sort a million numbers every time. t-digest (Ted Dunning, 2019) maintains a compressed representation using “centroids”, each centroid stores a mean and a weight (how many values it represents).

The key insight: centroids near the tails (p=0 and p=1) are kept small, giving high precision exactly where percentile queries need it. The p50 centroid might represent thousands of values (low precision, but p50 doesn’t need it). The p99.9 centroid might represent only 3 values (high precision, because p99.9 matters for SLA monitoring).

Memory usage is bounded: typically 100-500 centroids regardless of how many values you’ve ingested. A billion-value stream uses the same memory as a thousand-value stream.

KLL Sketch: Quantiles With Provable Error Bounds

t-digest is great in practice, but it doesn’t come with a formal worst-case guarantee. If you need to write in a paper or a compliance doc that “the 95th percentile estimate is within 1% of truth, I can prove it,” you want KLL instead.

The Core Idea, Told With Stairs

Imagine you’re collecting values in a bucket. When the bucket fills up, you can’t keep everything, so you play a little game: sort what you have, then keep every other element and throw the rest away. You pour the survivors into a second, bigger bucket. That bucket represents twice the information per slot, meaning each element now “stands in for” two of the originals.

Second bucket fills up. Same game. Sort it, keep every other, dump the survivors into a third bucket, which now represents four originals per slot. And so on, up the stairs.

At the top of the staircase, you’ve got a small sample that represents a huge stream. Each bucket remembers its “weight”: how many original items each stored value stands for. When you ask “what’s the 90th percentile,” the algorithm walks the stairs, weighting each sample appropriately, and gives you an answer.

That’s KLL (Karnin-Lang-Liberty, 2016) in plain English. Each “compactor” is one step on the staircase. The key property: total memory grows only logarithmically with how much data you stream. Double the input, add one step. Double again, add one more. You could stream a trillion items and the staircase would still fit in a kilobyte or two.

The Clever Wrinkle

The step “sort, keep every other element” has a subtle problem: if you always keep the even-indexed ones, you introduce bias: the lower half of every pair slightly dominates. KLL alternates: sometimes keep the evens, sometimes keep the odds. Over many compactions, the bias cancels out, and you get the formal guarantee: your rank error is bounded by ε with high probability, using memory that scales as O((1/ε) log² log(1/δ)).

For the default setting (k=200), expect ~1% rank error in a few kilobytes, for any input size you can reasonably imagine.

When to Pick KLL vs. t-digest

• t-digest: better tail accuracy (p99.9, p99.99) for SLA monitoring. Empirically excellent. Use when operators trust you.
• KLL: clean theoretical bounds. Use when a reviewer, auditor, or paper demands a proof.

DDSketch: Relative-Error Quantiles for Latency

The Core Idea, Told With a Ruler

Imagine a ruler for measuring fish. If your ruler has one-centimeter markings, it’s great for measuring small fish. A 4 cm guppy reads accurately. For a 4-meter tuna, the centimeter markings are overkill; the tuna is 400 cm whether you read it as 399 or 401, and nobody cares.

Now imagine the opposite ruler: the smallest marking is 10 cm. Great for tuna (400 cm ± 5 cm, no problem). But now the guppy reads as “about 0 cm” and you’ve completely lost it.

Real latency data is the fish problem. A web request can take 5 microseconds (database cache hit) or 5 minutes (deadlock on a cross-region replica). Both numbers matter. Neither ruler works for both.

DDSketch’s trick: build a ruler whose markings get wider as values get bigger. Small values get fine markings. Big values get coarse ones. The key property is that any two values within the same marking are guaranteed to be within a fixed percentage of each other, 1% for the default setting.

How It Works

Pick an error target (say, 1%). From that, compute a number called γ (gamma) slightly bigger than 1. Now lay out your “markings” (really, buckets) so that bucket i covers values from γⁱ to γⁱ⁺¹.

Any two values in the same bucket differ by at most a factor of γ. That’s where the 1% guarantee comes from: by construction, you can never mis-estimate a value by more than that fixed percentage, no matter how small or large it is.

Inserting a new value takes two operations: compute its log, divide by log(γ), round up. That’s the bucket index. Increment the bucket’s counter. Done.

Buckets are stored in a sparse map keyed by index, so empty buckets cost nothing. A real latency stream might only touch a few hundred of them in total, memory stays tiny.

DDSketch is what Datadog uses for latency percentiles, and once you see the ruler analogy it’s obvious why. It’s also the right tool for anything where you care about ratios rather than absolute differences: sizes in bytes, prices in dollars, response times in milliseconds. Anywhere “how much bigger is this than that” is a more natural question than “how much larger in absolute terms.”

HyperLogLog: Counting Distinct Things

“How many unique users hit the site today?” sounds easy until you realize you can’t fit a hundred million user IDs in a hash set on a phone. You need a way to estimate how many distinct things showed up without keeping the things themselves.

The Core Idea, Told With Coins

Flip a coin until you get tails. Count how many heads in a row you got first.

Most of the time you’ll get a short streak, zero, one, two heads. A streak of five heads is rare. A streak of ten is very rare. If I walk up to you and tell you “the longest streak I saw today was ten heads,” you can make a reasonable guess that I flipped a lot of coins. Roughly 2¹⁰ ≈ 1,000 of them. Not because every streak of ten corresponds to exactly a thousand flips, but because on average you need about that many tries before a ten-streak shows up.

HyperLogLog plays the exact same game with data. Every item you feed it gets hashed into a random-looking 64-bit number. The library tracks the longest run of leading zeros it has seen across all those numbers. A run of 20 zeros is rare enough that it strongly suggests you’ve processed around 2²⁰ ≈ a million unique items. A run of 30 suggests a billion.

That’s the whole trick. You never stored the items. You stored a single number: the longest streak you’ve seen. From that number alone, you can estimate how many distinct things walked past you.

Why One Counter Isn’t Enough

A single streak counter is wildly noisy. One really lucky hash could give you a 30-streak after only a hundred items, and now your estimate is off by a factor of ten million.

The fix is: use many counters and average them. Split the incoming hash into two pieces, a “bucket prefix” (the first few bits) and a “remainder.” The bucket prefix decides which counter gets updated. The remainder is where you count the leading-zero streak.

Hash an item.
  First 14 bits → pick one of 16,384 counters.
  Remaining bits → count the leading zeros, update that counter if longer.

Different items hit different counters. Each counter sees roughly total / 16,384 unique items and records its own longest streak. Averaging those 16,384 streaks is wildly more accurate than trusting any single one. Variance drops by a factor of the square root of the counter count, which for 16,384 counters is about 128×.

The default configuration uses 16,384 single-byte counters, for a grand total of 16 KB of memory, smaller than a single PNG icon on a web page. Error at that size is about 0.8%. That’s billions-of-events cardinality estimation in a buffer that fits in L1 cache.

Edge Cases

When a stream has very few items, most counters are still zero and the estimate is biased high. When it has astronomically many, there’s a different kind of bias at the top end. The implementation includes corrections from the Heule et al. “HyperLogLog in Practice” paper for both tails. You don’t need to think about them. They kick in automatically, and they’re why the library’s estimates are accurate at a thousand items and at a trillion items, not just at the sizes the original paper happened to test.

Count-Min Sketch: Counting How Often Things Happen

HyperLogLog tells you how many distinct things you saw. Count-Min Sketch (Cormode-Muthukrishnan, 2005) tells you how often a specific thing showed up: the “heavy hitters” problem.

The structure is a 2D table: depth rows by width columns. Each row has its own hash function. To increment, you hash the value with each row’s hash and increment the cell table[row * width + hash]. To query, you look up all depth cells and return the minimum.

table[0][h₀(x)] += 1
table[1][h₁(x)] += 1
table[2][h₂(x)] += 1
...
estimate(x) = min over all rows

Why the minimum? Each cell is an upper bound on the true count (other items hash there too and inflate it). The minimum is the tightest upper bound across rows. With width ε⁻¹ and depth log(δ⁻¹), the estimate is within ε·N of the true count with probability 1−δ.

The catch: it can overestimate but never underestimate. Perfect for “find me anything that appeared more than 1% of the time” queries. Wrong for “exactly how many times did X appear”, use a real hash table for that.

vizcrush ships an 8KB-default sketch (width=1024, depth=5) that handles millions of events per second on a single thread.

Reservoir Sampling: A Random Sample From a Stream

You have an infinite stream of values and you want to keep a uniformly random sample of size k, without ever knowing the total length in advance. Vitter’s Algorithm R from 1985 does this in three lines of logic:

For each new item (1-indexed count n):
  if the reservoir has fewer than k items:
    append the new item
  else:
    pick a random index j in [0, n)
    if j < k: replace reservoir[j] with the new item

The first k items go straight in. After that, item N is kept with probability k/N, and if it’s kept it replaces a uniformly random existing slot. The algebra works out so that at every point in time, every item seen so far has exactly k/N probability of being in the reservoir. No bias toward early or late items. No knowledge of N required.

vizcrush uses an XOR-shift PRNG (the same one we use for hash seeding) instead of a real RNG, because we want deterministic results in tests and reproducible bug reports. If you need cryptographic randomness for sampling, you’re solving a different problem and should not be using this.

The use case is simple: you’re streaming a billion events, you want to feed a 1,000-point scatter plot, and you want every event to have an equal chance of being shown. Reservoir sampling is the right call.

So far we’ve been talking about picking the right data to show. Before you can pick well, though, you often need to clean up what you have, sort it, normalize it, filter it, reshape it. That’s the boring but critical plumbing of the next chapter.

Sorting, Normalizing, Filtering, Reshaping: The Grunt Work

TL;DR: The unglamorous operations that show up in every chart pipeline: sort a million numbers, rescale to [0,1], clip to a viewport, switch to a log axis, align distributions. Each one gets a tight, boring, fast implementation, and the chapter explains the small tricks that make “boring” ten times faster than the obvious approach.

Every glamorous algorithm in this book sits on top of a pile of unglamorous ones. Before LTTB can pick the best points, someone has to hand it data that’s sorted by time. Before a heatmap can be color-mapped, someone has to normalize the counts to [0, 1]. Before a chart can zoom, someone has to filter out everything outside the viewport. Before a log-scaled axis can render, someone has to compute a million logarithms.

These are the operations that nobody talks about at conferences. They’re the plumbing under the bathroom floor. And, like plumbing, they’re invisible when they work and catastrophic when they don’t.

This chapter is about the plumbing.

Radix Sort: O(n) Beats O(n log n)

JavaScript’s Array.sort() uses TimSort, comparison-based, O(n log n). For numeric data, we can do better.

Radix sort processes numbers digit by digit, from least significant to most. No comparisons needed; just counting occurrences of each digit and redistributing. For 8-byte floats, that’s 8 passes of O(n) each, giving O(n) total.

There’s a fun trick for sorting floats: IEEE 754 doubles don’t sort correctly as raw bits because negative numbers use sign-and-magnitude representation. The fix is a two-line bit hack:

to_sortable(v):
  bits = raw 64-bit representation of v
  if sign bit is 1 (negative): flip all 64 bits
  else (positive):             flip only the sign bit

After this transformation, the bit patterns sort in the same order as the float values. Radix sort on the transformed bits, then invert the transformation to get the original values back.

Normalization

Min-max normalization scales values to [0, 1]:

normalized = (value - min) / (max - min)

Dead simple, but it shows up in every heatmap color-mapping, every scatter plot size encoding, every gradient computation. We pre-compute 1.0 / (max - min) and multiply instead of dividing in the loop. Division is ~10x slower than multiplication on most hardware.

Range Filtering

When the user zooms to a viewport [xMin, xMax], we need to extract just those points. For unsorted data, it’s a linear scan (O(n)). For sorted time-series (which is the common case), we binary search for the start and end indices (O(log n)) and return a slice. No copying at all.

Log and Power Transforms

A scatter plot of file sizes, request latencies, or cryptocurrency market caps will always look the same: a few huge values dominate the top, and 99% of the data is squashed into the bottom 5% of the chart. The fix is changing the axis scale instead of changing the data. But the chart library still has to compute log values for every point, often in JavaScript, often on every redraw.

vizcrush moves that work into the compute layer: log_transform, ln_transform, log10_transform, and power_transform each map an input array to an output array in a single pass. The result is a Float64Array you hand to your chart library on a linear axis: the math has already happened.

The algorithm is obvious. The only trick worth mentioning: for an arbitrary-base logarithm, compute ln(base) once outside the loop and multiply by its reciprocal per element. Natural log is expensive, and computing it once instead of once per element is the kind of change that costs nothing and pays back forever.

Non-positive inputs become NaN, not an error and not a silent zero. NaN propagates through the chart library as a gap, which is exactly what should happen for log(−1).

Power transforms take the same single-pass shape: one tight loop, no allocations beyond the output array.

Quantile Normalization

The Core Idea, Told With Class Rankings

Imagine two classrooms in two different countries, each with 30 students, each with their own English test scores. Class A has scores from 60 to 95. Class B has scores from 40 to 85. A student with a score of 78 in Class A is the “10th best” in that class. A student with a score of 72 in Class B is also the 10th best.

If you want to fairly compare “how good is each student relative to their classmates,” raw scores won’t work: the tests weren’t the same and the grading wasn’t the same. What does work is comparing ranks. “10th best” means the same thing in both classes, regardless of scale.

Quantile normalization takes this idea and goes one step further. It doesn’t just compare ranks. It rewrites every score so that students of the same rank across classes end up with the same number. The 10th-best student in both classes gets the average of the two 10th-best scores. The 1st-best students get the average of the two top scores. And so on.

After this rewrite, the two classes have identical distributions (the same set of sorted scores), but each student’s position in their own class is preserved. You’ve made the two classes directly comparable without distorting their internal rankings.

Why Anyone Cares

This trick was invented for gene-expression data. Biologists run experiments where each “column” is a different biological sample (a person, a tissue, a mouse) and each “row” is a different gene. The raw expression values can’t be compared directly across samples because each sample was measured on its own noisy scale. Quantile normalization makes them comparable by forcing every sample to share the same global distribution of values, while preserving the relative ordering of genes within each sample.

It’s narrow-sounding until you realize how often the same problem shows up elsewhere: A/B-test metrics from different buckets that had different baseline behavior, per-user activity data where user sizes differ wildly, time-series from sensors with different calibrations. Anywhere you want to say “let’s pretend these all came from the same underlying distribution and just compare ranks”, quantile normalization is your friend.

The Algorithm

1. Sort each column independently. Remember each value's original row index.
2. For each rank position r, compute the mean of all columns' values at that rank.
   Call the result rank_means[r].
3. Replace each value in each column with rank_means[r], where r is that
   value's rank within its own column.

Three steps, one slightly slippery catch: NaN handling. If a column has missing values, they sort to the end (not the start, which would corrupt everything), and any rank position where even one column has a NaN produces a NaN rank mean. Missing data stays missing. The algorithm doesn’t silently invent numbers for it.

Complexity is O(n_cols · n_rows · log n_rows) for the sorts, O(n_cols · n_rows) for the rest, and the memory footprint is one additional matrix the same size as the input. For a 50,000 × 100 expression matrix (a realistic gene-expression dataset), WASM runs this in tens of milliseconds. Pure JavaScript takes seconds. This is exactly the kind of operation that makes the WASM port pay for itself on the first call.

So that’s the two-dimensional world taken care of. What happens when your data has a third dimension, a camera moving around it, and a visibility problem that doesn’t exist in flat space? That’s what the next chapter is about.

3D: Voxels, Octrees, Frustum Culling

TL;DR: Three dimensions break things in two ways. Memory scales worse, and the camera introduces a whole new problem: visibility. This chapter is the 3D dialect of everything you learned in chapters 6 and 7, plus a new trick (frustum culling) that only makes sense once you have a camera.

The moment your data has a third dimension, every technique from earlier in the book needs a parallel story. A 2D scatter plot becomes a point cloud. A quadtree becomes an octree. Binning becomes voxelization. And there’s one entirely new concept (frustum culling) that doesn’t exist in 2D because 2D has no camera.

This chapter covers the three 3D primitives vizcrush ships: bin3d (voxel binning), octree (3D spatial index), and frustum (camera-aware culling).

Why 3D Is Harder

Two reasons, both of them about scaling.

Memory scales as N³, not N². A 128×128 grid has 16K cells. A 128×128×128 voxel grid has 2 million cells. Double the resolution and you 8× the memory. You cannot naively scale up what worked in 2D. The bookkeeping cost grows faster than the data.

Visibility is a first-class concern. In a 2D chart, everything you care about is on the canvas at once. In a 3D scene, the camera looks at a specific region; everything outside that region is invisible and should never be processed. Skipping invisible points before you do anything else is often a bigger win than any algorithmic cleverness on the visible ones.

Voxel Binning: 2D Binning in Three Dimensions

A voxel is a 3D pixel. You define a bounding box, divide it into an x_bins × y_bins × z_bins grid, and count how many input points fall into each cell. The output is a flat array of length x_bins · y_bins · z_bins, indexed linearly as grid[z · y_bins · x_bins + y · x_bins + x].

The algorithm is exactly what you’d expect from 2D binning with one more axis:

for each point (px, py, pz):
    xi = floor((px − x_min) / x_range · x_bins)
    yi = floor((py − y_min) / y_range · y_bins)
    zi = floor((pz − z_min) / z_range · z_bins)
    clamp xi, yi, zi to [0, bins − 1]
    grid[zi · y_bins · x_bins + yi · x_bins + xi] += 1

The output also carries the bin edges (x_edges, y_edges, z_edges) appended to the grid so the caller can label axes or map voxels back to world coordinates without recomputing anything.

Auto-ranging: pass NaN for any min/max and vizcrush derives it from the data in one pass. This is the common case; you rarely know your exact data bounds ahead of time, and forcing the caller to compute them just to call the binning function is exactly the kind of friction that makes people write their own janky version.

Use cases:

• LIDAR / point clouds: reduce millions of raw scan points to a dense voxel grid you can ray-march or render directly.
• Molecular dynamics: count how often atoms occupy each region of a simulation box to produce a probability density.
• Medical imaging: re-bin scattered samples onto a uniform grid for volume rendering.
• Any 3D histogram: the “scatter plots lie” problem from chapter 6 is twice as bad in 3D because depth overlap masks density even more aggressively.

A warning about resolution: picking x_bins = y_bins = z_bins = 256 sounds reasonable until you realize you just allocated 16 million f64 cells (128 MB of RAM) before putting a single point in. For 3D, start with resolution around 64 per axis and go up only if you have a reason. A 64³ grid is 262K cells, fits comfortably in L2 cache, and renders beautifully for most point clouds.

Octree: Quadtree’s 3D Cousin

An octree splits 3D space into eight children at each level instead of four. The naming convention matches the sign of each axis in the parent’s local frame:

Child 0: −x −y −z    Child 4: −x −y +z
Child 1: +x −y −z    Child 5: +x −y +z
Child 2: −x +y −z    Child 6: −x +y +z
Child 3: +x +y −z    Child 7: +x +y +z

Everything you learned about quadtrees in chapter 7 translates directly: build cost is O(n log n), range queries prune whole subtrees, nearest-neighbor queries walk the closer child first. The math is the same, there’s just one more axis to partition on.

vizcrush’s octree uses two implementation details worth calling out:

Arena allocation, not pointer trees. Every node lives in a single flat Vec<OctNode>. Children are referenced by index into the arena, not by pointer. This is a huge cache-locality win: walking the tree touches contiguous memory, and serializing the whole tree is a single memory copy. It’s also essential for passing the tree across the WASM boundary without rebuilding it on each side.

Partitioned index array. Points aren’t duplicated into leaf nodes. Instead, each node owns a [start, end) slice into a single shared indices: Vec<u32> array. Building the tree is essentially an in-place partial sort. Points get reordered so that every subtree’s points live in a contiguous run. Queries iterate those runs directly. No pointer chasing, no scattered reads.

The thresholds: leaf nodes hold up to 64 points, maximum depth is 12. These come from the same reasoning as the quadtree: 64 is small enough that brute-force comparison in a leaf is cheap, and depth 12 (roughly 68 billion maximum cells) is way more than anyone needs but keeps pathological clustered inputs from blowing the stack.

When to use an octree vs. a voxel grid:

• Voxel grid: you want to know density per region. Memory is fixed by resolution, not by point count.
• Octree: you want to query points: “all points inside this box,” “k nearest to this cursor ray,” “which points are inside this selection volume.” Memory is fixed by point count, not by resolution.

If your use case is “show me a heatmap of density” → voxel. If it’s “let the user hover and pick” → octree. If it’s both, build both, they’re cheap and they solve different problems.

Frustum Culling: Don’t Process What You Can’t See

A camera in 3D doesn’t see all of space. It sees a frustum: a truncated pyramid with six boundary planes (near, far, left, right, top, bottom). Anything outside those six planes is invisible, and any compute you spend on it is wasted.

Frustum culling is the operation of testing each point against those six planes and keeping only the ones inside. It’s conceptually the 3D equivalent of range filtering from chapter 9, but with an important twist: the “range” is defined by six tilted planes rather than two axis-aligned intervals, and the planes change every time the camera moves.

The Math

A plane in 3D is defined by the equation a·x + b·y + c·z + d = 0. The vector (a, b, c) is the plane’s normal direction, and d shifts the plane along that normal. For any point P = (px, py, pz), the signed distance from the plane is:

dist(P) = a · px + b · py + c · pz + d

If we choose the convention that positive distance means “inside the frustum,” then a point is visible if and only if dist(P) > 0 for all six planes.

visible(P) = (dist_near(P)   > 0)
           & (dist_far(P)    > 0)
           & (dist_left(P)   > 0)
           & (dist_right(P)  > 0)
           & (dist_top(P)    > 0)
           & (dist_bottom(P) > 0)

Six multiply-adds per plane, six planes, one AND reduction. 36 operations per point to answer “is this visible”. That’s vastly cheaper than computing what color it should be or where it lands in screen space.

Why Plane Normalization Matters

When you first construct a plane from (a, b, c, d), the (a, b, c) vector is probably not unit-length; it depends on how the plane was derived (projection matrices, cross products, etc). Non-normalized planes still give you the correct sign for point-in-frustum tests, but the magnitude of dist(P) isn’t a real distance; it’s scaled by |(a, b, c)|.

For a pure visibility test, this doesn’t matter. But the moment you want to compute “how far is this point from the near plane” (for LOD selection, depth sorting, or progressive rendering), you need real distances. vizcrush normalizes planes at construction by dividing all four coefficients by |(a, b, c)|, so dist(P) is always the true signed distance in world units.

|n|   = sqrt(a² + b² + c²)
a, b, c, d ← a/|n|, b/|n|, c/|n|, d/|n|

Do it once at construction. You never think about it again.

Integrating Frustum Culling with the Octree

The real power combo: prune the octree against the frustum before you walk it.

At each octree node, test the node’s bounding box against the frustum. Three outcomes:

1. Box fully inside frustum → return every point in the subtree, no further testing.
2. Box fully outside frustum → skip the entire subtree.
3. Box straddles the frustum → recurse into children and test individually.

A scene with 10 million points but a camera looking at 5% of the space typically processes maybe 500K points. That’s a 20× win before you do anything else.

This is how every serious 3D viewer, Potree, deck.gl, Three.js with BVH, stays interactive on huge point clouds. It’s not the renderer that’s fast. It’s the culling that makes the renderer’s job small enough to be fast.

3D Decision Cheat Sheet

Everything else you know from earlier chapters still applies, the 3D primitives compose with downsampling (LTTB over the visible set after culling), with streaming (voxel updates on a rolling window), and with normalization (remap z into a color channel). The dimensional jump changes the structures, not the ideas.

Speaking of “the ideas”, most of this book has been about computing things faster or smaller. The next chapter is different. It’s about computing things that explain themselves, so that a language model (or a curious human) can reason about your data without staring at ten million raw values.

AI-Assisted Viz: Anomaly, Shape, Auto-Config

TL;DR: These four utilities turn a dataset into structured answers that a language model can actually consume. No neural networks, no embeddings. Just classical statistics wired up as the hooks LLMs use to reason about data they’ll never see.

Everything else in this book is about making the browser fast enough to show a million points. This chapter is about something different: using statistics to figure out what you should be showing in the first place.

The word “AI” in the chapter title is only half accurate. There’s no neural network. No model weights. No inference server. These four tools (anomaly detection, shape fingerprints, auto-configuration, summarization) are classical statistics applied well. The “AI” label only fits because they’re the hooks that real language models can call to reason about your data without ever seeing the raw numbers.

An LLM doesn’t need to ingest two million sensor readings. It needs to ask “are there any anomalies?” and “what does this shape look like?” and “how should I render it?” and get back three small, structured answers. That’s what this chapter’s primitives provide.

Anomaly Detection: The Modified Z-Score

The textbook anomaly detector uses the mean and standard deviation: flag anything more than 3σ from the mean. This is popular, widely implemented, and wrong for real data.

The problem is that a single huge outlier pulls both the mean and the standard deviation toward itself. The very values you’re trying to detect contaminate the statistics you’re using to detect them. You end up raising the threshold just enough to hide the thing you wanted to find.

The fix is to use robust statistics: ones that don’t get dragged around by extreme values. vizcrush uses the Modified Z-Score, based on Median Absolute Deviation (MAD):

median       = median(y)
abs_devs[i]  = |y[i] − median|
MAD          = median(abs_devs)
z[i]         = 0.6745 · (y[i] − median) / MAD

The median is immune to outliers. You can add a value of a billion to a dataset and the median barely moves. The MAD is the median of the absolute deviations, which inherits the same immunity. The constant 0.6745 rescales MAD so that, for normally-distributed data, |z| > 3 corresponds to the same threshold as the classical 3σ test. You get the familiar cutoff without the contamination problem.

detect_anomalies(y, sensitivity) returns an interleaved array [index₀, value₀, score₀, index₁, value₁, score₁, ...]. One entry per flagged point. The interleaving is deliberate: a single Float64Array is the zero-copy format for the WASM → JS boundary, and the caller can iterate three at a time without parsing anything.

When to use it:

• Highlighting spikes on a live monitoring chart before the eye can spot them
• Feeding an LLM agent a short list of “interesting moments” instead of the full series
• Triggering alerts on streaming data where the mean is untrustworthy

Edge case worth knowing: if MAD is zero (meaning more than half your data points share the exact same value), the function returns an empty list rather than dividing by zero. This is the right call. A dataset with that structure has no meaningful dispersion to measure anomalies against, and silently producing Infinity scores would be worse than returning nothing.

Shape Vectors: “Find Me Charts That Look Like This”

Suppose you have a thousand dashboards and a user says “I want to find all the ones that look like this CPU spike I’m staring at.” You can’t do a visual comparison against a thousand charts. You need a way to turn each chart’s shape into a number, and then compare those numbers.

That’s what compute_shape_vector(y, dimensions) does. It produces a fixed-length vector of numbers that captures the visual character of a time series. The vector encodes the overall trend, the volatility, the spike ratio, the autocorrelation at a few different lags, and a normalized histogram of the distribution. Two series that look alike end up with similar vectors. Two that look different end up far apart.

The vectors can be compared with cosine similarity, used as keys in a nearest-neighbor search, or fed to a clustering algorithm to group dashboards automatically. It’s a cheap, explainable alternative to “run a deep learning model over every chart.”

The components (conceptually):

1. Normalized histogram. Rescale the data to [0, 1], bin it, record the density per bin. Captures distribution shape: bimodal, skewed, uniform.
2. Trend coefficient. Fit a line to the data and record the slope. Captures “is this going up or down.”
3. Volatility. Standard deviation of the first differences. Captures “how jagged is this.”
4. Spike ratio. Count of points outside 2σ divided by total count. Captures “how many outliers.”
5. Autocorrelation at multiple lags. Correlation between y[t] and y[t − lag] for a few different lag values. Captures periodicity and smoothness.

All of these are cheap to compute in a single or double pass. None of them require keeping the original data around once the vector is built: the vector is the summary.

Use case: a user pins a shape they care about. You store its vector. Every new chart gets its vector computed once, compared against the pinned vectors, and tagged. The UI can then say “this looks like the CPU spike pattern” without any human labeling.

The dimensions parameter: pass how many dimensions you want in the output. Higher dimensions capture more nuance but make comparison slower and more sensitive to noise. For most dashboards, 16–32 dimensions is plenty: enough to distinguish archetypal shapes, small enough to compare thousands of vectors in milliseconds.

Summarize: Statistics Small Enough to Fit in a Prompt

summarize(y) computes the classical set of summary statistics: count, mean, standard deviation, min, max, median, skewness, kurtosis. All in one pass over sorted data (the sort is O(n log n); everything else is O(n) once).

Two of those are worth pausing on.

Skewness measures asymmetry. Positive skew means the right tail is longer (big outliers on the high side). Negative means the left tail is longer. Zero is symmetric. The formula:

skewness = (1/n) · Σ ((yᵢ − mean) / stddev)³

Cubing preserves sign (negative deviations stay negative), so the sum cancels for a symmetric distribution and grows for a one-sided one.

Kurtosis measures tail weight. High kurtosis means a distribution with fatter tails than a Gaussian: more extreme values than you’d expect. Low kurtosis means thinner tails. The formula:

kurtosis = (1/n) · Σ ((yᵢ − mean) / stddev)⁴

Fourth power because we want to emphasize extremes and we don’t care about sign at that point.

Together, skewness and kurtosis answer the question “is this data normally distributed or not?” A Gaussian has skewness 0 and excess kurtosis 0. Financial returns have high kurtosis. The 2008 crash was possible. Sensor noise tends to be low-kurtosis and symmetric. Knowing these two numbers tells you whether the standard statistical machinery will even work on your data.

Why this matters for AI: the entire summary fits in a dozen floats. You can put it in a prompt. An LLM sees “count 1.2M, mean 47.3, stddev 18.1, skew 2.4, kurtosis 9.1” and instantly knows this is a long-tailed distribution. No scatter plot required. The model’s recommendation becomes better because its input became smaller and more structured.

Auto-Configuration: Let the Library Decide

auto_config(x, y, target_width) looks at a dataset and returns a structured recommendation for how to render it. The output includes:

• Algorithm: which downsampler to use (lttb, minmax_lttb, etc.)
• Target points: how many points to output (usually a small multiple of display width)
• Bin resolution, for scatter / heatmap data
• Spatial index, quadtree, kdtree, or hashgrid
• Streaming config, window size + update interval, if the data looks live
• Estimated speedup, a rough factor compared to rendering every point
• Reasoning, a short string explaining why these choices

The decision logic is simple and transparent, no opaque model. It checks basic properties of the data (is x monotonically increasing → time series; is the spread huge → probably needs log scale; how many points are outside 2σ → spikey or smooth) and applies rules from the decision chapter you’ll find later in this book.

The reasoning string is the interesting part. When the recommendation is wrong, a human, or a language model, can read the reasoning and override it. When the recommendation is right, the reasoning is documentation: “I picked MinMaxLTTB because 3.2% of your points are more than 2σ from the mean, so spike preservation matters for this dataset.”

Two ways to use this:

1. Interactive: call auto_config when the user first loads a dataset. Apply the result as defaults. Let them adjust.
2. Agent-driven: an LLM calls auto_config through the MCP server (see the library docs), gets the structured result plus the reasoning, and either applies it or asks the user “I’d recommend MinMaxLTTB because of spikes, OK?”

Either way, the goal is the same: move the decision-making logic out of “someone had to read the docs and pick correctly” into the library itself. Every step away from hand-tuning is a step toward dashboards that just work.

The Throughline

None of this is glamorous machine learning. No embeddings, no fine-tuning, no GPUs training anything. Just statistics applied with some taste and wired up to a structured output format that tools, human or otherwise, can actually consume.

That’s the point. The “AI-readiness” of a library isn’t about adding a model. It’s about making your data’s properties queryable in a form a model can reason about without hallucinating. A summary with skewness and kurtosis beats “here’s 2 million raw samples, figure it out.” A shape vector beats “look at this screenshot.” An auto-config result beats “read the docs and choose.”

Build the primitives that answer specific questions, and the rest of the AI story writes itself.

That’s the last of the algorithm chapters. You’ve now met every tool in the vizcrush toolbox. The next two chapters are about choosing, how to pick the right tool for a specific problem, and how to actually wire it into a real codebase.

Picking the Right Tool

TL;DR: A flowchart from “what are you trying to do?” to “here’s the function to call.” Bookmark this chapter. You’ll come back to it more than any other.

You’ve now seen everything vizcrush ships: four downsamplers, three binning strategies, four spatial indexes, seven streaming sketches, five transforms, three 3D primitives, four AI utilities. That’s a lot of functions. The question this chapter answers is: which one do you actually reach for, when?

If you’re already an expert and you just want the flowchart, skip ahead to the end of the chapter. If you want to understand why the flowchart looks the way it does, read straight through, every branch in the tree maps back to a decision we made for a real reason.

The Three Questions

Before you pick any algorithm, answer three questions about your data. The answer to each one eliminates whole categories of tools.

Q1: Is the data one-shot or streaming?

• One-shot: you have a fixed dataset on disk or in memory, and you want to render it. You’re going to call a function once per zoom level. → Use the batch primitives.
• Streaming: data arrives continuously and you want the chart to keep up at 60fps. You’ll call a function many times per second. → Use the streaming primitives (chapter 8) and appendAndDownsample specifically.

Q2: What’s the underlying structure?

• Time series: x-axis is time or some other monotonic ordering. You want to see shape over time.
• Scatter: x and y are independent features. You want to see clusters and density.
• Spatial: x and y (and maybe z) are positions. You want to query points by region or proximity.
• Categorical / distribution: you care about “how many of each” or “what does the distribution look like” rather than individual points.

Q3: What matters most to the viewer?

• Overall shape (trends, seasonality)
• Extreme values (spikes, crashes, anomalies)
• Exact extremes (compliance, audit trails)
• Density (clusters, overlaps)
• Specific points (hover, selection)

Three questions, three axes. Every algorithm in vizcrush lives at some point in that space.

The Decision Tree

Is it streaming?
├─ YES  → ch 8. Use StreamingStats + appendAndDownsample.
│         ├─ Need variance/std_dev?          → Welford
│         ├─ Need percentiles?
│         │    ├─ p50-p95, general use       → t-digest
│         │    ├─ p99.9+ for SLA             → DDSketch (relative-error)
│         │    └─ Need formal error bound    → KLL
│         ├─ Need cardinality (distinct)?    → HyperLogLog
│         ├─ Need frequency (heavy hitters)? → Count-Min Sketch
│         └─ Need a uniform random sample?   → Reservoir sampling
│
└─ NO (one-shot)
   │
   What's the structure?
   │
   ├─ Time series? (x is monotonic)
   │    │
   │    What matters most?
   │    ├─ Smooth shape, no spikes           → LTTB
   │    ├─ Spikes matter (finance, IoT)      → MinMaxLTTB
   │    ├─ Exact extremes required           → M4
   │    └─ Speed over quality                → LTOB
   │
   ├─ Scatter / density?
   │    │
   │    What dimension?
   │    ├─ 2D density heatmap
   │    │    ├─ Fast                         → bin2d (rectangular)
   │    │    └─ Visually uniform             → hexbin
   │    ├─ 1D histogram                      → bin1d
   │    └─ 3D density / point cloud          → bin3d (voxel)
   │
   ├─ Spatial queries?
   │    │
   │    What query pattern?
   │    ├─ Viewport / range (zoom/pan)       → quadtree
   │    ├─ Hover picking (k-NN)              → quadtree kNN
   │    ├─ Fixed-radius brush / lasso        → hash grid
   │    ├─ Sort-based, cache-friendly build  → Morton codes
   │    └─ Any of the above, but 3D          → octree (+ frustum)
   │
   ├─ 3D with a camera?
   │    └─ Cull before anything else         → frustum (+ octree)
   │
   └─ Distribution / categorical?
        ├─ Histogram of values               → bin1d
        ├─ Multiple distributions side by side → quantile_normalize
        └─ Just summary statistics           → summarize

Keep this diagram nearby. Almost every question someone asks me reduces to a path through it.

The Canonical Scenarios

Let’s walk through seven real situations and see how the tree resolves them. These are the actual questions I get asked.

Scenario 1: “My dashboard has 10M rows of server CPU usage over a month. It freezes.”

Questions: one-shot (fixed dataset), time series (x is timestamp), shape matters (trends), maybe spikes too (for incidents).

Path: one-shot → time series → spikes matter → MinMaxLTTB with target_points ≈ chart_width (so ~1920 for a desktop chart, ~800 for a mobile one).

Why not LTTB: server CPU has spikes you care about (load surges, GC pauses). LTTB smooths them.

Why not M4: overkill. M4 returns 4× the points for not much visual gain at this density.

Scenario 2: “I have a live feed of 1,000 metrics/second and I want to show the last 10 minutes with running stats.”

Questions: streaming (live feed), time series, shape + summary stats.

Path: streaming → StreamingStats for running mean/std/min/max, appendAndDownsample for the chart.

Why not “just use LTTB every second”: you’d re-scan the entire buffer on every tick. appendAndDownsample is specifically designed to avoid the double pass, it combines the append, the eviction, and the LTTB in a single operation with no intermediate allocation.

Add a t-digest or DDSketch alongside if the user also wants to see p50/p95/p99 bands.

Scenario 3: “My 500K-point scatter plot shows 10 dots when it should show clusters.”

Questions: one-shot, scatter, density matters.

Path: one-shot → scatter → density → bin2d at 128×128. Color-map the result to show the density field.

If the clusters look square/blocky: switch to hexbin. Hexagons have equidistant neighbors, so round clusters render as round clusters instead of square ones.

Also: build a quadtree on the raw points if users will want to pick individual points after seeing the heatmap.

Scenario 4: “I need to show the 99th percentile of API latency over the last hour, and it has to stay accurate as new data streams in.”

Questions: streaming, percentiles, specifically p99, latency (log-scaled).

Path: streaming → percentiles → p99 on latency → DDSketch with α = 0.01 (1% relative error).

Why DDSketch and not t-digest: DDSketch’s relative-error guarantee means “1% of the true value” whether latency is 5ms or 5 minutes. t-digest’s accuracy is absolute and degrades on log-scaled data.

Why not KLL: KLL is great when you need a formal worst-case bound for a paper. For SLA monitoring, DDSketch’s practical behavior is better.

Scenario 5: “I want to show which genes are ‘unique’ in a 100-sample × 50K-gene expression matrix.”

Questions: one-shot, matrix (multi-distribution), shape comparison.

Path: one-shot → multiple distributions → quantile_normalize to make samples comparable, then compute per-row summary statistics for ranking.

Why not plain normalization: min-max scales to the same range but doesn’t align shapes. Two samples with identical sort orders can have completely different min-max normalized values. Quantile normalization forces both samples into the same distribution shape, so a rank of “5th highest” maps to the same value in every sample.

Scenario 6: “I have a LIDAR scan with 20M points and I’m rendering a first-person fly-through.”

Questions: one-shot dataset but interactive rendering, 3D, camera-aware.

Path: one-shot → 3D → camera-aware → frustum cull first (prune to the ~5% visible on each frame), then query an octree built over the surviving points for hover picking and distance-based LOD.

Don’t voxelize unless you also want a density heatmap. The raw points are what you want to render; culling + indexing is what keeps it fast.

Memory tip: build the octree once. Rebuild it only when the underlying point cloud changes (on new LIDAR sweeps), not on every frame.

Scenario 7: “An LLM agent is asking about my data and I don’t want to stream 10MB of raw numbers to it.”

Questions: one-shot, any shape, needs compressed representation.

Path: call summarize and compute_shape_vector (chapter 11). Feed the resulting 8–32 floats to the model instead of the raw data. If the model asks “are there anomalies,” also call detect_anomalies and pass the flagged indices.

Also consider auto_config: it’ll give the model a structured recommendation it can apply or override. The reasoning field is a short human-readable string, ideal for prompt injection into the assistant’s context.

Parameter Guidance

The decision tree tells you which algorithm. The next question is with what parameters. Short answers here; longer notes inline throughout the book.

Downsampling

• target_points, start with chart width in pixels (~1920 for desktop, ~800 for mobile). Double it for retina displays if you want extra margin for sharp lines.
• MinMaxLTTB bucket multiplier, the default 4× is almost always right. Higher preserves more spikes but slows things down.

Binning

• bins, target a density where most cells have >5 points. For 500K points, 128×128 is a good start. For 50K, try 64×64.
• 3D bins, start at 64³ or less. Memory scales cubically.

Spatial indexes

• Quadtree leaf size, 64. Don’t touch unless you’re building a specialized tool.
• Hash grid cell size, match your typical query radius. Slightly larger is safer than slightly smaller.

Streaming sketches

• HyperLogLog precision p, 14 gives ~0.8% error using 16KB. Drop to 12 for ~1.6% error using 4KB if memory is tight.
• KLL k, 200 gives ~1% rank error. 400 for ~0.5%.
• DDSketch α, 0.01 (1% relative error) for most latency work. 0.005 if you’re obsessive.
• t-digest compression, 100 is fine for most dashboards. 200 if you care about p99.9.
• Count-Min Sketch (width, depth), (1024, 5) is the default and handles “what are the top-k heavy hitters” for millions of distinct keys.

AI utilities

• Anomaly sensitivity, 3.0 (modified z-score equivalent of 3σ). Raise to 4–5 to reduce false positives; lower to 2 to catch softer anomalies.
• Shape vector dimensions, 16 to 32. More dimensions capture more nuance but make comparison noisier.

A Two-Line Summary

Most problems in visualization fall into one of two buckets:

1. Too much data for the screen, downsample, bin, or sketch.
2. Too much data for the query, index it (quadtree, kd-tree, hash grid, octree) or cull it (frustum, range filter).

Almost every algorithm in this book is a specific answer to one of those two problems. When you’re stuck, start with that framing and pick the tool that matches your structure.

Everything else is parameters.

The flowchart tells you what. The next chapter tells you how, with real code, real imports, and real API calls you can copy and paste.

Using vizcrush: A Tour in Code

TL;DR: Every vizcrush function, shown in the smallest possible TypeScript example. Copy, paste, ship. If you only read one chapter in this book, make it this one, and bookmark it.

The rest of the book explains what vizcrush does and why it does it the way it does. This chapter is the bridge: how to actually call these functions from a real project. One code example per algorithm family, in TypeScript, with the understanding that every example assumes pnpm install, a bundler that handles ESM, and a modern browser.

If you want to follow along, the library lives at github.com/debug-diary-1/vizcrush.

Install

vizcrush is published as a set of narrow packages. Install only what you need:

pnpm add @vizcrush/core @vizcrush/downsample

Add more as you go, @vizcrush/bin, @vizcrush/aggregate, @vizcrush/spatial, @vizcrush/spatial3d, @vizcrush/bin3d, @vizcrush/transform, @vizcrush/ai, @vizcrush/react.

Step 1: Initialize and Check Capabilities

Before you call anything, ask vizcrush what it can do. The init() call probes the environment and picks the backend: the Rust/WASM build when WebAssembly is available, the pure-JS core otherwise.

import { init } from "@vizcrush/core";

const ctx = await init();

console.log(ctx.backend);
// 'wasm' | 'js'

console.log(ctx.capabilities);
// { webgpu: true, wasm: true, wasmSimd: true, sharedArrayBuffer: false }

You only call init() once per page. The returned ctx object is shared by every other vizcrush call. The capabilities object is raw environment probes for your own diagnostics; only wasm decides the backend. Per call, small inputs run on the JS core regardless (the WASM boundary has a fixed cost that tiny arrays never pay back), and you can force a path with { backend: "js" | "wasm" } if you’re benchmarking. The reader will not notice any of this.

Nothing else in this chapter needs you to look at ctx again. It’s always there in the background doing the right thing.

Step 2: Downsample a Time Series

You have a million points. Your chart can show 1920. lttb does the selection:

import { lttb } from "@vizcrush/downsample";

// x and y are Float64Arrays, same length
const { x: outX, y: outY } = await lttb(x, y, 1920);

// outX and outY have length 1920, feed them to your chart

That’s the whole API. Pass typed arrays in, get typed arrays out. If x is already sorted (which it is for time series), you don’t need to do anything else.

For spiky data, swap in minMaxLttb:

import { minMaxLttb } from "@vizcrush/downsample";

const { x: outX, y: outY } = await minMaxLttb(x, y, 1920);

Same signature, different guarantees. See chapter 5 for when to pick which.

Sync variant: if you’re in a hot render loop and can’t await, use lttbSync. It runs the JS core directly and returns the result immediately.

Step 3: Build a 2D Density Heatmap

bin2d is for scatter plots where overlapping dots hide your clusters:

import { bin2d } from "@vizcrush/bin";

const result = await bin2d(x, y, {
  xBins: 128,
  yBins: 128,
  // omit xRange / yRange to auto-detect from the data
});

// result.grid is a Uint32Array of length 128 * 128
// result.xEdges and result.yEdges give you the bin boundaries
// result.maxCount is the largest bin count, handy for color scaling

Render result.grid as a heatmap, map count to color, done. You now see density instead of a blob of dots.

For visually isotropic clusters, use hexbin instead:

import { hexbin } from "@vizcrush/bin";

const hexes = await hexbin(x, y, 10); // radius
// hexes is an array of { cx, cy, count }, one per non-empty hex

Step 4: Index Points for Hover and Range Queries

Interactive charts need to answer “what point is under the cursor” in under a millisecond. Build a quadtree once, query it many times:

import { buildQuadtree, queryRange, queryNearest } from "@vizcrush/spatial";

const tree = await buildQuadtree(x, y);

// Range query: all point indices in a viewport
const indices = queryRange(tree, {
  xMin: 100,
  yMin: 200,
  xMax: 500,
  yMax: 600,
});

// k-NN query: the 5 closest points to the cursor
const nearest = queryNearest(tree, cursorX, cursorY, 5);

// indices / nearest are Uint32Arrays, map them back to your data

For fixed-radius queries (brush selection, find-neighbors-within-r), use a hash grid instead:

import { buildHashGrid, hashGridQueryRadius } from "@vizcrush/spatial";

const grid = await buildHashGrid(x, y, 20); // cell size
const neighbors = hashGridQueryRadius(grid, cursorX, cursorY, 15);

The right index depends on the query pattern, see the decision tree in chapter 12.

Step 5: Stream Live Data Without Re-scanning

For a live dashboard, two things matter: rolling stats that update in O(1), and a chart that stays smooth as new batches arrive.

import { streamingStats, appendAndDownsample } from "@vizcrush/aggregate";

const stats = streamingStats(20_000); // rolling window of 20K values

function onNewBatch(newValues: Float64Array) {
  // Push the batch into the rolling window (stats.pushBatch under the
  // hood — stats.push(value) is the one-at-a-time variant) and
  // downsample to 1920 display points in one call
  const { x, y } = appendAndDownsample(stats, newValues, 1920);

  // Running min / max / mean / stddev, updated in O(1) per value
  console.log(stats.mean, stats.stdDev, stats.min, stats.max);

  chart.update({ x, y });
}

The point of appendAndDownsample is that it’s a single call: it feeds the batch into the accumulator (append and evict inside the rolling window) and hands back a render-ready { x, y } pair. You never manage the buffer yourself. 60fps streaming charts become boring to build.

Step 6: Percentiles on a Stream

Live SLA monitoring, p50, p95, p99 over the rolling window:

import { DDSketch } from "@vizcrush/aggregate";

const sketch = new DDSketch(0.01); // 1% relative error

function onLatencyBatch(latencies: Float64Array) {
  for (const lat of latencies) sketch.add(lat);

  const p50 = sketch.quantile(0.5);
  const p95 = sketch.quantile(0.95);
  const p99 = sketch.quantile(0.99);

  chart.updateBands({ p50, p95, p99 });
}

DDSketch gives relative-error guarantees, ideal when latencies span microseconds to seconds. For general percentile work without the relative-error story, KllSketch is the same API.

For cardinality (“how many distinct users”), use HyperLogLog the same way:

import { HyperLogLog } from "@vizcrush/aggregate";

const hll = new HyperLogLog(14); // precision → ~0.8% error
// add() takes numbers — hash string ids to a number first
for (const userId of newUserIds) hll.add(userId);
console.log("approx distinct:", hll.estimate());

Step 7: Transforms Before Plotting

Log-scaling a million points in JavaScript is the kind of thing that makes a chart feel sluggish. Do it in WASM instead:

import { log10Transform, normalize } from "@vizcrush/transform";

const logY = await log10Transform(y);     // NaN for non-positive values
const scaled = await normalize(logY);     // min-max to [0, 1]

chart.setData({ x, y: scaled });

Every transform takes a Float64Array, resolves to a Float64Array. No in-place mutation, no surprises.

Step 8: 3D Point Clouds

Voxel binning for a density field:

import { bin3d } from "@vizcrush/bin3d";

const voxels = await bin3d(x, y, z, {
  xBins: 64,
  yBins: 64,
  zBins: 64,
});
// voxels.grid is a Uint32Array of length 64 * 64 * 64
// voxels.xEdges / yEdges / zEdges and voxels.maxCount round it out

Octree + frustum culling for an interactive viewer:

import { buildOctree, queryRange3d } from "@vizcrush/spatial3d";
import { frustumCullSync } from "@vizcrush/spatial3d";

const octree = await buildOctree(x, y, z);

// Every frame, given the current camera (column-major 4x4 MVP):
function renderFrame(viewProj: Float64Array) {
  // Step 1: reject everything outside the frustum (cheap pre-filter)
  const visibleIndices = frustumCullSync(x, y, z, viewProj);

  // Step 2: render the surviving points, use octree for hover picking
  renderer.draw(visibleIndices);
}

The culling step is the win. You can render 20M-point scans at 60fps because 95% of the points never reach the GPU pipeline.

Step 9: Let the Library Decide

If you’re not sure which algorithm to pick, or you’re building an agent that needs to decide for itself, call autoOptimize:

import { autoOptimize } from "@vizcrush/ai";

const recommendation = autoOptimize(x, y, 1920); // screen width in px

console.log(recommendation);
// {
//   algorithm: "minmax_lttb",
//   targetPoints: 3840,
//   binResolution: null,
//   spatialIndex: "none",
//   streaming: null,
//   estimatedSpeedup: 65.1,
//   reasoning: "Spiky time-series detected (spike ratio 14.2). Using MinMax+LTTB to preserve extrema in 250000 points → 3840."
// }

The reasoning field is the interesting part, it’s a short human-readable string explaining why the recommendation is what it is. You can show it to a user, feed it to an LLM, or ignore it. Everything else is a drop-in parameter for the actual algorithm calls.

For a quick summary of any dataset:

import { summarize, detectAnomalies, computeShapeVector } from "@vizcrush/ai";

const summary = summarize(x, y);
// { trend, trendSlope, range, distribution, spikeCount, anomalyCount, summary }
// distribution is { mean, median, stddev, skewness }

const anomalies = detectAnomalies(y, 3.0);
// array of { index, value, zScore, type: "spike" | "dip" | "shift" }

const shape = computeShapeVector(y, 16);
// Float64Array of 16 features, cosine-similar to "similar" charts

Step 10: React Hook

For React users, @vizcrush/react wraps the common patterns:

import { useDownsample } from "@vizcrush/react";

function Chart({ rawX, rawY }: { rawX: Float64Array; rawY: Float64Array }) {
  const { data, loading } = useDownsample(rawX, rawY, {
    algorithm: "lttb", // or "minmax_lttb" | "m4"
    threshold: 1920,
  });

  if (loading) return <div>Downsampling…</div>;
  return <LineChart data={data} />;
}

The hook handles:

• Re-running the downsample when inputs change
• Ignoring stale results when a newer call resolves first
• Reporting loading, error, and an elapsed timing you can log

If you’re not using React, the vanilla API above is all you need, the hook is a convenience, not a requirement.

The Mental Model

Every vizcrush function follows the same shape:

1. Input: one or more Float64Array (or equivalent typed array). Contiguous memory, no objects, no per-point allocations.
2. Output: another Float64Array (or a small handle for things like spatial indexes). Same contract.
3. No rendering: vizcrush never touches the DOM, the canvas, or the chart library. You pass the output to your renderer of choice.

Once you internalize this, every new algorithm in the library fits the same template. bin1d takes an array, returns an array. lttb takes arrays, returns arrays. computeShapeVector takes an array, returns an array. You never wrestle with object models or configuration schemas, typed arrays all the way down.

The next time you find yourself about to write for (let i = 0; i < 500_000; i++) in JavaScript, stop. Check this book’s decision chapter. There’s probably a vizcrush function that already did the work, faster, without blocking your main thread.

The remaining three chapters are for the curious: how vizcrush stays working when the hardware doesn’t cooperate, the performance lessons I learned the hard way, and, for anyone who wants to dig into the math, a reference of the formulas behind the algorithms.

Why It Doesn’t Break (Fallback Strategy)

TL;DR: Every algorithm ships in two implementations: Rust/WASM and pure JavaScript. Detection happens once at init, selection happens per call, and the user never sees which one ran.

A library that only works when everything goes right is a demo, not a library. vizcrush has to work on a five-year-old Chromebook, a brand-new Mac on stable release, and a headless Node environment with no browser at all. One codebase, wildly different execution environments, zero tolerance for “sorry, this browser isn’t supported.”

Here’s the rule: vizcrush must always work. Not “work if you have the latest Chrome.” Always.

The fallback cascade:

WebAssembly available? → Use the Rust/WASM build
    ↓ no
Pure JavaScript core   → Always works, everywhere

Two tiers, not four. An earlier design imagined a taller ladder (WebGPU on top, separate SIMD and scalar WASM rungs below), but chapter 3 already told you what the measurements did to that idea: no WebGPU path was ever wired, and the SIMD build turned out byte-identical to the scalar one. The honest architecture is two engines with full parity.

Detection is done once at init time. You can’t just check typeof WebAssembly; we validate a minimal WASM module to confirm the environment will actually compile one. After that, selection is per call: below a small data size the kernel prefers the JS core regardless, because crossing the WASM boundary has a fixed cost that tiny inputs never repay.

The JavaScript core is feature-complete and parity-tested: every algorithm has a pure JS implementation verified in CI to produce the same output as the WASM build. And calling it a “fallback” undersells it. On 1M-point LTTB it runs in roughly 1.5–6ms depending on the engine, and in Firefox it’s the faster path. Either way, both engines are orders of magnitude better than the 850ms+ your chart library would burn trying to render all million points.

The WASM modules only load if they exist. In development (before running the WASM build script), the JS core kicks in silently. No 404 errors, no console spam. I spent an annoying amount of time making the bundler not try to statically analyze the WASM import path.

Robustness is about never falling off the cliff. Performance is about how fast you run on the flat part. Those are two different disciplines, and the next chapter covers the tricks for the second one.

Making It Fast (Performance Lessons)

TL;DR: Four lessons from building this library: memory layout beats clever algorithms, allocation in the hot path is the silent killer, every WASM boundary crossing has a cost, and most benchmarks are lying to you.

There’s a famous quote (I can’t remember who said it) that goes something like: “make it work, make it right, make it fast, in that order.” Good advice. What they don’t tell you is that “make it fast” is actually four separate jobs, and if you skip any one of them, the other three don’t matter.

The four jobs are: respect the CPU cache, respect the garbage collector, respect the boundary between JavaScript and WASM, and respect your own benchmarking setup. This chapter is what I learned about each one while shipping vizcrush, mostly by doing it wrong first.

Memory Layout Is Everything

// This is slow:
const points = [{ x: 1, y: 2 }, { x: 3, y: 4 }, ...];

Each object is a separate heap allocation. When you iterate, the CPU fetches a cache line (64 bytes), finds a pointer, follows it to some random memory location, fetches another cache line to read the actual data, then does it all again for the next point. Cache miss after cache miss.

// This is fast:
const x = new Float64Array([1, 3, ...]);
const y = new Float64Array([2, 4, ...]);

Contiguous memory. The CPU fetches a cache line and gets 8 sequential values. Prefetchers predict the linear access pattern and start loading the next cache line before you need it. Night and day difference.

This is why vizcrush uses typed arrays everywhere and returns interleaved data across the WASM boundary: [x0, y0, x1, y1, ...]. One contiguous allocation, zero garbage collector involvement, maximum cache utilization.

Avoid Allocation in the Hot Path

Every new Object() in a tight loop is work for the garbage collector later. When the GC runs, it pauses all JavaScript execution: potentially for several milliseconds. During a pan gesture that’s firing 60 times per second, a GC pause is a visible stutter.

vizcrush pre-allocates output buffers and returns a single typed array. No intermediate objects. No per-point allocations. The GC has nothing to clean up.

The WASM Boundary Isn’t Free

Calling from JavaScript into WASM has overhead: about 10-50 nanoseconds per call depending on the arguments. If you’re calling a WASM function a million times in a loop, the boundary overhead adds up.

vizcrush avoids this by doing all the looping inside WASM. You call lttb(x, y, threshold) once, it does the entire million-point computation internally, and returns the result. One boundary crossing, not a million.

Benchmark the Right Way

Microbenchmarks are liars. I’ve seen benchmarks show WASM as slower than JS because:

1. No warmup: The first run includes JIT compilation time
2. Too few runs: A single outlier (GC pause, OS scheduler preemption) skews the result
3. Mean instead of median: One 50ms outlier in 100 runs of 1ms each makes the mean 1.5ms

vizcrush’s benchmark suite does 5 warmup runs (discarded), then 100 measured runs, reports the median and p95. The CI pipeline compares against a committed baseline and fails on >15% regression (configurable for noisy environments; laptops get 25%).

That’s the end of the narrative part of the book. The last chapter is a reference: the math behind every algorithm, for people who want to verify the implementation or modify it. If you came here to use vizcrush, you can close the tab with a clean conscience. If you came here to understand, the math chapter is the dessert.

The Math Behind It All

TL;DR: The formulas every earlier chapter referred to but didn’t spell out. Safe to skip. Useful if you want to modify the algorithms, verify them, or argue about them on the internet.

This chapter is the appendix, not the finale. If the earlier chapters gave you the story, this chapter is the receipts, the actual formulas behind each algorithm, written out for anyone who wants to check the homework.

You don’t need any of this to use vizcrush. You might want some of it if you plan to port an algorithm, tune a parameter past its default, or write your own variant. Skim the headings and dip in where you’re curious.

Triangle Area (Cross Product Form)

Given three points (x₁,y₁), (x₂,y₂), (x₃,y₃):

Area = ½ |x₁(y₂ - y₃) + x₂(y₃ - y₁) + x₃(y₁ - y₂)|

This is the absolute value of half the cross product of vectors (P₂-P₁) and (P₃-P₁). No square roots or trigonometry. Three multiplications, five additions, one absolute value. This is what makes LTTB fast: the inner loop does almost no work per candidate.

Welford’s Derivation

Starting from the definition of sample mean:

μₙ = (1/n) Σ xᵢ

When xₙ₊₁ arrives:

μₙ₊₁ = μₙ + (xₙ₊₁ - μₙ) / (n+1)

For the sum of squared deviations M₂:

M₂,ₙ₊₁ = M₂,ₙ + (xₙ₊₁ - μₙ)(xₙ₊₁ - μₙ₊₁)

The product (xₙ₊₁ - μₙ)(xₙ₊₁ - μₙ₊₁) is always non-negative (both factors have the same sign because μₙ₊₁ is between μₙ and xₙ₊₁). This means M₂ increases monotonically, so there are no cancellation errors.

Sample variance: σ² = M₂ / (n - 1)

Morton Code Bit Interleaving

To interleave the bits of two 16-bit integers into one 32-bit integer:

expand_bits(v):
    v = (v | v << 8) & 0x00FF00FF
    v = (v | v << 4) & 0x0F0F0F0F
    v = (v | v << 2) & 0x33333333
    v = (v | v << 1) & 0x55555555
    return v

morton(x, y) = expand_bits(x) | (expand_bits(y) << 1)

Each step “spreads” the bits apart with zeros between them. The final OR merges x bits into even positions and y bits into odd positions.

IEEE 754 Sortable Encoding

A 64-bit float in IEEE 754:

[sign: 1 bit] [exponent: 11 bits] [mantissa: 52 bits]

Positive floats already sort correctly as unsigned integers (exponent ordering, mantissa breaks ties). Negative floats sort backwards (more negative = larger bit pattern).

The fix:

• Positive: flip the sign bit (so positives sort after negatives)
• Negative: flip ALL bits (inverting the backward ordering)

This is an order-preserving bijection from f64 to u64, enabling radix sort on floats.

t-digest Scale Function

The compression function that controls centroid size:

k(q) = (δ / 2π) · arcsin(2q - 1) + δ/2

where q ∈ [0,1] is the quantile position and δ is the compression parameter.

The derivative dk/dq approaches infinity at q=0 and q=1, forcing centroids near the tails to be small (high resolution). At q=0.5, the derivative is at its minimum, allowing large centroids (low resolution where it doesn’t matter).

DDSketch Logarithmic Buckets

Given a relative-accuracy target α ∈ (0, 1):

γ     = (1 + α) / (1 - α)
b(v)  = ⌈ln(v) / ln(γ)⌉      ← bucket index for value v

Any value v falls into bucket b(v), and any two values in the same bucket differ by at most a factor of γ. That’s where the relative-error guarantee comes from: pick α, derive γ, every quantile estimate is within α of the truth on a multiplicative scale.

HyperLogLog Estimate

For m = 2ᵖ registers with values M[0..m):

raw = α_m · m² / Σᵢ 2^(-M[i])

where α_m is a bias-correction constant (0.7213/(1+1.079/m) for large m).

Small-range correction: when many registers are still zero, linear counting is more accurate: m · ln(m / zeros). Large-range correction uses the Heule et al. empirical bias table (not reproduced here).

Hexagonal Grid Coordinates

A regular hexagon with radius r:

Width between parallel edges:  r√3
Distance between hex centers:  2r (horizontal), r√3 (vertical)
Row offset:                    r for odd rows, 0 for even rows

Point-to-hex assignment:

row = round((y - y_min) / (r√3))
offset = r if row is odd, else 0
col = round((x - x_min - offset) / (2r))

This is an approximation. The exact nearest-hex computation requires checking the three closest candidates. The round-based approximation is correct for >98% of points and the error only occurs at hex boundaries where it doesn’t matter for visualization.

Colophon

That’s the book. If you read this far, you understand everything inside vizcrush, not just how to use it, but why it works the way it does. Go build something fast.

This edition is built with mdBook from Markdown sources at github.com/debug-diary-1/vizcrush-book. Corrections, clarifications, and translations are welcome, open a PR.

The free edition is released under CC BY-NC 4.0. Non-commercial redistribution, translation, and quotation are encouraged. A commercial edition with additional material is planned.

— The vizcrush team, 2026