crossfilter3

A streaming-first, zero-copy analytics engine for the browser.

Crossfilter3 is built on top of crossfilter2, already the fastest way to filter large datasets client-side. This fork turns it into a complete dashboard runtime — data streams in as Arrow IPC, decodes and filters inside a Web Worker (main thread never blocks), WASM accelerates the hot filter scan, and partial snapshots render the UI progressively before the download even finishes.

What crossfilter3 adds to crossfilter2

Layer	What changed	Why it matters
Ingest	Columnar Arrow IPC streaming with batch coalescing (default 64K rows), multi-source lookup joins, projection/rename/type-coercion at ingest, Proxy-backed lazy row arrays that defer materialization via COLUMNAR_BATCH_KEY	Data goes from Cube.dev (or any Arrow source) straight into crossfilter's sorted indexes without ever building intermediate row objects — a 10K-row dataset allocates zero row objects until a panel actually reads one
Filtering	Inline WASM module (no external .wasm file) with two scan paths: filterInU32 for small target sets (k ≤ 4, n ≤ 1K) and markFilterInU32 with dense lookup for larger sets; automatic regex extraction of d => d.field and function(d){ return d.field } into string paths for WASM routing	Filter scans stay in linear WASM memory instead of JS object traversal — see performance estimates below
Lazy encoded path	Uint32 code encoding per dimension (code 0 = null, 1..n = distinct values), 2x-amortized codes buffer growth on append, incremental codeCounts updates, filterRange target-codes caching, groupAll O(1) fast path, scratch buffer reuse across filter cycles	Appends are O(batch) not O(n log n) — a 10K append into 90K existing records skips the full re-sort and re-reduce
Aggregation	Declarative KPIs (count, sum, avg, avgNonZero), declarative groups with time bucketing (minute/hour/day/week/month), split-field groups for nested aggregates keyed by a secondary dimension, incremental group updates on append	One config object replaces dozens of imperative dimension().group().reduce() chains
Worker runtime	createStreamingDashboardWorker owns fetch → decode → filter → reduce → postMessage with Transferable typed-array buffers (zero-copy back to main thread); createDashboardRuntime for synchronous fallback; single query() round-trip returns filters + snapshot + paged rows + optional rowCount and bounds	The main thread only renders; structured-clone overhead is eliminated for the largest payloads (column arrays)
Query model	Declarative filters (exact, in, range), isolatedFilters for within-group filtering without affecting global state, rowSets for multiple named row slices per query, bounds queries for min/max, ad-hoc groups queries	One postMessage round-trip replaces what would otherwise be 4-8 separate calls
Progressive UI	Partial snapshot emission during streaming load (throttled at 250ms default), separate fetch-percent and rows-loaded progress events (throttled at 100ms)	Charts and KPIs appear within seconds even on million-row datasets
Live mutation	append() slots new rows into existing lazy indexes incrementally; removeFiltered() rebuilds codeCounts safely and re-filters	Dashboards stay live without full rebuild
Instance extensions	cf.allFilteredIndexes(), cf.isElementFiltered(index), cf.takeColumns(indexes, fields), cf.configureRuntime() / cf.runtimeInfo() for per-instance WASM control	Columnar extraction and filter introspection without materializing rows
The original crossfilter API (cf.dimension(), group.all(), etc.) is fully preserved — everything above is additive.

Performance and memory estimates

These are analytical estimates based on the architecture — not synthetic benchmarks. Actual results depend on dataset shape, dimension cardinality, and browser.

Filter scan throughput (WASM vs JS)

The WASM module operates on flat Uint32Array codes in linear memory. The JS fallback (denseLookupMatches) builds a marks array and iterates in JS. Both do the same work — the difference is memory access pattern and JIT overhead.

Operation	JS fallback	WASM (markFilterInU32)	Speedup estimate
filterIn on 100K rows, 50 target values	~2-4ms	~0.5-1.5ms	~2-3x
filterIn on 1M rows, 50 target values	~20-40ms	~5-15ms	~2-4x
filterExact on 100K rows	~0.5-1ms	~0.2-0.5ms	~2x

The small-target path (filterInU32, k ≤ 4) uses a tight O(n*k) nested loop — effective for filterExact and small filterIn sets where the marks array setup cost would dominate.

Memory footprint per dimension

Component	Size for N rows	Example (100K rows)
codes (Uint32Array, 2x capacity)	4 × 2N bytes	800 KB
codeCounts (Uint32Array)	4 × cardinality bytes	4 KB (1000 distinct)
codeToValue (Array)	~50 × cardinality bytes	50 KB
selected (Uint8Array)	N bytes	100 KB
Filter bitmask (Uint8/16/32)	1-4 × N bytes	100-400 KB
Total per dimension	~6-10 bytes/row	~1 MB

Adding a dimension costs ~6-10 bytes per row. The 32-dimension limit (bitmask width) means worst-case overhead is ~320 bytes/row.

Row materialization savings

In upstream crossfilter, every record is a JS object from the start. In this fork, columnar ingest creates Proxy-backed arrays — rows materialize only on access.

Scenario	Upstream (all rows as objects)	This fork (columnar + lazy)	Savings
10K rows, 20 fields, 50 visible	~5 MB (10K objects × ~500 bytes)	~25 KB (50 objects) + columnar arrays already in memory	~5 MB heap, ~10K fewer GC objects
100K rows, 20 fields, 50 visible	~50 MB	~25 KB + columnar	~50 MB heap
1M rows, 20 fields, 100 visible	~500 MB	~50 KB + columnar	~500 MB heap

The columnar arrays themselves (one typed/string array per field) are the same size either way — the saving is entirely in not creating N row objects with N × fields property slots.

Append performance (lazy path)

Operation	Upstream	This fork (lazy encoded)	Why
Append 10K rows to 90K	O(n log n) re-sort + full reduce	O(m) codes extension + incremental codeCounts	Codes buffer grows 2x amortized; existing sorted indexes untouched
groupAll after append	O(n) full scan	O(1) mark update	Singleton groups skip sorted-key rebuild

Installation

npm install crossfilter3 apache-arrow

The streaming worker needs UMD bundles of both libraries available at public HTTP URLs. In a Next.js project, copy them into public/ with a postinstall script:

// package.json
{
"scripts": {
"postinstall": "mkdir -p public/vendor && cp node_modules/crossfilter3/crossfilter.js public/vendor/ && cp node_modules/apache-arrow/Arrow.es2015.min.js public/vendor/"
}
}

Then reference them as absolute paths:

construntime=awaitcrossfilter.createStreamingDashboardWorker({crossfilterUrl: '/vendor/crossfilter.js',arrowRuntimeUrl: '/vendor/Arrow.es2015.min.js',// ...});

Quick start

importcrossfilterfrom'crossfilter3';construntime=awaitcrossfilter.createStreamingDashboardWorker({crossfilterUrl: '/vendor/crossfilter.js',arrowRuntimeUrl: '/vendor/Arrow.es2015.min.js',wasm: true,emitSnapshots: true,batchCoalesceRows: 65536,// Declare filterable fields (use post-projection names)dimensions: ['event','country','region','time'],// Global aggregate metricskpis: [{id: 'count',field: 'count',op: 'sum'}],// Pre-computed group-by aggregations for chartsgroups: [{id: 'byEvent',field: 'event',metrics: [{id: 'count',field: 'count',op: 'sum'}]},{id: 'byCountry',field: 'country',metrics: [{id: 'count',field: 'count',op: 'sum'}]},{id: 'timeline',field: 'time',bucket: {type: 'timeBucket',granularity: 'month'},metrics: [{id: 'count',field: 'count',op: 'sum'}],},],// Arrow IPC source (fetched inside the worker)sources: [{id: 'primary',role: 'base',dataUrl: '/api/cube',dataFetchInit: {method: 'POST',headers: {'Content-Type': 'application/json'},body: JSON.stringify({format: 'arrow',query: {dimensions: ['events.country','events.event','events.region'],measures: ['events.count'],timeDimensions: [{dimension: 'events.timestamp',granularity: 'month'}],timezone: 'UTC',limit: 1000000,},}),},projection: {rename: {'events.count': 'count','events__count': 'count','events__timestamp_month': 'time','events.event': 'event','events.country': 'country','events.region': 'region',},transforms: {count: 'number',time: 'timestampMs',},},}],});

Progress and streaming snapshots

While data loads, the worker emits progress and partial snapshots so the UI can render progressively:

runtime.on('progress',(progress)=>{console.log(progress.status,progress.load.rowsLoaded,'rows');console.log(progress.fetch.percent,'% downloaded');});runtime.on('snapshot',({ snapshot })=>{renderCharts(snapshot.groups);renderKpis(snapshot.kpis);});awaitruntime.ready;

Querying with filters

After load, send declarative filters and get back pre-computed aggregations plus paged row data in a single worker round-trip:

constresult=awaitruntime.query({filters: {country: {type: 'in',values: ['US','UK']},time: {type: 'range',range: [startMs,endMs]},},rows: {sortBy: 'time',direction: 'top',limit: 50,offset: 0,fields: ['event','country','region','time','count'],},});result.snapshot.kpis;// { count: 54321 }result.snapshot.groups.byEvent;// [{ key: 'click', value: { count: 30000 }}, ...]result.snapshot.groups.timeline;// [{ key: 1704067200000, value: { count: 12000 }}, ...]result.rows;// columnar row data for the table

Filter types

// Exact match{type: 'exact',value: 'click'}// Set membership{type: 'in',values: ['click','view','purchase']}// Range (inclusive lower, exclusive upper){type: 'range',range: [startMs,endMs]}// Clear filter on a fieldnull

Live data mutation

Append rows or remove filtered records without rebuilding the runtime:

awaitruntime.append([{event: 'click',country: 'US',region: 'CA',time: Date.now(),count: 1},]);awaitruntime.removeFiltered('excluded');

Synchronous fallback (no worker)

For smaller datasets or environments where workers are unavailable:

importcrossfilterfrom'crossfilter3';import{tableFromIPC}from'apache-arrow';constbuffer=awaitfetch('/data/result.arrow').then(r=>r.arrayBuffer());consttable=tableFromIPC(newUint8Array(buffer));construntime=crossfilter.createDashboardRuntime({
table,wasm: true,dimensions: ['country','event','time'],groups: [{id: 'byCountry',field: 'country',metrics: [{id: 'count',op: 'count'}]},],kpis: [{id: 'total',op: 'count'}],});constsnapshot=runtime.snapshot({country: {type: 'in',values: ['US']},});

Classic crossfilter API

The original crossfilter API is still fully available:

importcrossfilterfrom'crossfilter3';constcf=crossfilter(records);constcountry=cf.dimension('country');country.filterIn(['US','UK']);constgroup=country.group().reduceCount();console.log(group.all());

Configuration reference

createStreamingDashboardWorker(options)

Option	Type	Description
crossfilterUrl	string	URL to the UMD build (crossfilter.js)
arrowRuntimeUrl	string	URL to Apache Arrow UMD (Arrow.es2015.min.js)
sources	Array	Arrow IPC data sources (see below)
dimensions	string[]	Field names to create filterable dimensions on
groups	Array	Declarative group-by specs
kpis	Array	Global aggregate metric specs
wasm	boolean	Enable WASM-accelerated filter scans (default true)
emitSnapshots	boolean	Emit partial snapshots during streaming load
batchCoalesceRows	number	Buffer this many rows before flushing to the runtime (default 65536)
progressThrottleMs	number	Min interval between progress events (default 100)
snapshotThrottleMs	number	Min interval between snapshot events (default 250)
workerFactory	() => Worker	Custom worker factory (skips importScripts entirely)

Source spec

Field	Type	Description
id	string	Unique source identifier
role	'base' | 'lookup'	One base source required; lookups are joined in the worker
dataUrl	string	URL to fetch the Arrow IPC stream
dataFetchInit	RequestInit	Fetch options (method, headers, body)
arrowBuffer	ArrayBuffer	Pre-loaded Arrow buffer (alternative to dataUrl)
projection.rename	Record<string, string>	Rename Arrow columns to internal field names
projection.transforms	Record<string, string>	Type coercion: 'timestampMs', 'number', 'constantOne'

Metric spec

Field	Type	Description
id	string	Key in the result (snapshot.kpis[id], group.value[id])
field	string	Column to aggregate (not needed for 'count')
op	string	'count', 'sum', 'avg', 'avgNonZero'

Async runtime methods

Method	Returns	Description
runtime.ready	Promise<Progress>	Resolves when all data is loaded
runtime.on(event, fn)	() => void	Subscribe to 'progress', 'snapshot', 'error'
runtime.query(request)	Promise<{ snapshot, rows }>	Apply filters, return aggregations + row page
runtime.snapshot(filters)	Promise<Snapshot>	Aggregations only, no row data
runtime.rows(query)	Promise<RowResult>	Paged row data only
runtime.updateFilters(filters)	Promise	Update filters without reading results
runtime.append(records)	Promise<number>	Add rows, returns new dataset size
runtime.createGroup(spec)	Promise<string>	Add a group at runtime, returns its ID
runtime.disposeGroup(id)	Promise<void>	Remove a dynamically created group
runtime.removeFiltered(selection)	Promise<number>	Remove 'included' or 'excluded' rows
runtime.bounds(request)	Promise	Get min/max for fields
runtime.groups(request)	Promise	Ad-hoc group queries
runtime.dispose()	Promise	Terminate the worker

Architecture

Arrow IPC source (Cube.dev, file, etc.)
|
v HTTP streaming response
Web Worker (owns fetch, decode, crossfilter instance)
| Apache Arrow RecordBatchReader
| Incremental batch append with projection/rename/transforms
| WASM-accelerated encoded filter scans
v
Declarative filters in, snapshots + row slices out
|
v postMessage with Transferable buffers
Main thread (renders UI only)

Development

npm install
npm test# vitest + eslint
npm run build # rollup -> crossfilter.js + crossfilter.min.js
npm run benchmark

License

Apache-2.0. Based on the original crossfilter by Mike Bostock and Jason Davies.