BranchesTags

Repository files navigation

zarr-node

CILicense: MIT

Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.

Features

  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion

Install

Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):

@i4sea:registry=https://npm.pkg.github.com

Then install:

npm install @i4sea/zarr-node

For S3 support, install the peer dependency:

npm install @aws-sdk/client-s3

Quick Start

Read an array from the filesystem

import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);

Integer dtypes (int64 / uint64)

Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array or BigUint64Array. Their elements are bigint, not number. Coerce with Number(value) when you need a plain number — this is safe for epoch-seconds up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those ranges precision is lost.

consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number

Read from HTTP

import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();

Read from S3

import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();

Connection pooling and prewarming

S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:

conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});

maxSockets (default 128, keep-alive on) caps how many chunk fetches run in parallel — raise the read concurrency and keep maxSockets >= concurrency so a many-chunk read finishes in one wave instead of several. Run the reader in the same AWS region as the bucket — that, not the library, dominates latency.

Groups and multi-array reads

import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);

Bounding memory

Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk count. By default a single get() holds at most maxInFlightBytes (256 MiB) of decoded chunk data at once and copies each chunk into the output as it arrives, so peak memory stays predictable even on arrays with large chunks.

// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});

Compressed point-slices pay full-chunk cost. Selecting a single (lat, lon) from a blosc/gzip/zlib array still downloads and decompresses the entire chunk covering that point — partial decode isn't possible for these codecs. The cost is per chunk, not per element, so a wide selection over a chunked axis decodes one full chunk per step. maxInFlightBytes bounds how many of those decode concurrently; a MemoryCache avoids re-decoding chunks across repeated reads.

readMultiple shares one budget across all arrays, so reading many compressed arrays at once stays bounded by a single ceiling rather than arrays × concurrency × chunkSize.

Any read whose materialized output would exceed largeReadWarningBytes (512 MiB) — whether a full-array get() or a large slice — logs a one-line console.warn. Set it to Infinity to silence.

Sizing maxInFlightBytes from a RAM limit

Peak bytes a single in-flight chunk holds while being processed:

peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size per in-flight chunk; a compressed little-endian array holds 2×.

To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the process baseline and keep a safety margin:

maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB — remembering the read output buffer is allocated on top of the in-flight budget. maxInFlightBytes caps the combined decoded footprint regardless of concurrency or chunk size, so it is the binding knob for memory safety.

Caching

import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});

Eviction and cache sizing

After each write, CachedStore evicts the oldest entries by file modification time (least-recently-written — reads do not refresh an entry's eviction priority) so that store's cache stays at or below maxSizeBytes. A non-positive or non-finite maxSizeBytes is rejected at construction.

The limit is scoped per store, not per directory: each CachedStore keeps its entries under cacheDir/<hash(storeId)> and evicts only there. Several stores sharing one cacheDir can therefore use up to N × maxSizeBytes in total. For stores without a derivable identity (anything other than S3/HTTP, e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache subdirectory is created on every process start and stale ones are never evicted.

Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is ever evicted — every chunk fetched from the inner store is written to cacheDir and stays there, so sustained reads over a large dataset will eventually fill the disk (or the pod's ephemeral-storage limit, evicting the pod). Constructing a CachedStore without maxSizeBytes logs a console.warn for this reason; only omit it when the working set is known to fit on disk.

Sizing guidance:

  • Size for the hot working set, not the whole dataset — e.g. the chunks covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and reads fetch chunks concurrently (default concurrency 50), so usage can transiently exceed maxSizeBytes by roughly the read concurrency × chunk size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably below the container's ephemeral-storage limit (or mount a dedicated volume for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched repeatedly); if the hit rate is low, grow the limit or narrow the access pattern.

Shared metadata cache

open/openGroup/openArray accept a metadataCache implementing the async Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup, .zattrs) are served read-through: first open fetches from the store and caches; later opens — in the same process or, with Redis, on any pod — skip the store entirely. Entries are cached without TTL (datasets are immutable per path). A cache error or unavailable backend falls back to the store, so reads never fail because of the cache.

In-process:

import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });

Shared across pods via Redis (requires the optional ioredis peer dependency — npm install ioredis):

import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });

RedisCache also accepts a connection URL directly (new RedisCache("redis://..."), with optional ioredis options as a second argument); the client is then created lazily on first use. Passing a pre-configured client is preferred — with a bare URL, ioredis defaults apply and commands issued while Redis is unreachable can stall before the store fallback kicks in.

Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is derived automatically for S3Store and HTTPStore; for any other store you must pass an explicit storeId, otherwise open throws immediately (preventing silent per-pod key divergence):

awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});

Observability hooks

Every layer accepts an optional per-instance observability object — no global registry. The same object can be passed to multiple layers; each layer fires only the events it owns:

import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });

A throwing (or rejecting) handler is swallowed and never breaks a read. When no hooks are registered there is zero overhead — payload objects are not even allocated.

Offloading decompression (worker threads)

Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks the event loop for the whole decode — degrading the latency of every other request in a shared API pod. gzip/zlib already run on the libuv threadpool and are unaffected.

Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is offloadable (currently Blosc) and whose compressed size is at least minBytes are decoded on a worker thread; everything else decodes inline as before. Create one pool per process, reuse it across reads, and call terminate() on shutdown (idle workers keep the process alive).

import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();

The threshold is on the compressed size (known before decode). Use onChunkDecoded (above) to measure decodeMs with and without the pool and calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts runs that A/B and also reports event-loop lag.

Reference filesystem (kerchunk)

import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();

Spatial lookups (GridIndex)

@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j) on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it is loaded once and queried many times — each query is pure CPU.

import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);

For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:

import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });

Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs can't be trusted.

Polygon reads (readPolygon)

readPolygon streams — one time step at a time — only the cells geometrically inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a bounding-box block, so each backing chunk is fetched/decompressed at most once (chunks typically span the full time axis and are reused across steps via a shared MemoryCache), and peak memory stays bounded to ~one time slice regardless of the time extent. Aggregation is the caller's concern — you get the raw in-polygon values.

import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}

resolvePolygonCells(arr, opts) returns the time-invariant selection (cells + bbox + stride) without reading values — cells[k] aligns with step.values[k]. Three coordinate layouts are supported: { kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid } (curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured points). Set maxCells to cap huge selections with a clamped uniform stride (reported as selection.stride; no default cap). A runnable example lives in examples/read-polygon.ts.

Requirements

  • Node.js >= 22
  • ESM only ("type": "module")

API

Top-level functions

FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)

All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }. metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a content-versioned storeId so obsolete versions' keys expire from a shared cache instead of accumulating forever (omit ⇒ no expiry).

Store backends

ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest

Caching

ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)

Data classes

ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads

Spatial

ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)

Contributing

See CONTRIBUTING.md.

License

MIT

About

Read-only Zarr v2 array reader for Node.js

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
zarr-node
CILicense: MIT
Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.
Features
  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion
Install
Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):
@i4sea:registry=https://npm.pkg.github.com

Then install:
npm install @i4sea/zarr-node
For S3 support, install the peer dependency:
npm install @aws-sdk/client-s3
Quick Start
Read an array from the filesystem
import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);
Integer dtypes (int64 / uint64)
Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array
or BigUint64Array. Their elements are bigint, not number. Coerce with
Number(value) when you need a plain number — this is safe for epoch-seconds
up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those
ranges precision is lost.
consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number
Read from HTTP
import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();
Read from S3
import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();
Connection pooling and prewarming
S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:
conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});
maxSockets (default 128, keep-alive on) caps how many chunk fetches run in
parallel — raise the read concurrency and keep maxSockets >= concurrency so a
many-chunk read finishes in one wave instead of several. Run the reader in the
same AWS region as the bucket — that, not the library, dominates latency.
Groups and multi-array reads
import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);
Bounding memory
Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk
count. By default a single get() holds at most maxInFlightBytes (256 MiB) of
decoded chunk data at once and copies each chunk into the output as it arrives,
so peak memory stays predictable even on arrays with large chunks.
// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});
Compressed point-slices pay full-chunk cost. Selecting a single
(lat, lon) from a blosc/gzip/zlib array still downloads and
decompresses the entire chunk covering that point — partial decode isn't
possible for these codecs. The cost is per chunk, not per element, so a wide
selection over a chunked axis decodes one full chunk per step. maxInFlightBytes
bounds how many of those decode concurrently; a MemoryCache avoids
re-decoding chunks across repeated reads.
readMultiple shares one budget across all arrays, so reading many
compressed arrays at once stays bounded by a single ceiling rather than
arrays × concurrency × chunkSize.
Any read whose materialized output would exceed largeReadWarningBytes
(512 MiB) — whether a full-array get() or a large slice — logs a one-line
console.warn. Set it to Infinity to silence.
Sizing maxInFlightBytes from a RAM limit
Peak bytes a single in-flight chunk holds while being processed:
peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size
per in-flight chunk; a compressed little-endian array holds 2×.
To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the
process baseline and keep a safety margin:
maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and
runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB —
remembering the read output buffer is allocated on top of the in-flight
budget. maxInFlightBytes caps the combined decoded footprint regardless of
concurrency or chunk size, so it is the binding knob for memory safety.
Caching
import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});
Eviction and cache sizing
After each write, CachedStore evicts the oldest entries by file modification
time (least-recently-written — reads do not refresh an entry's eviction
priority) so that store's cache stays at or below maxSizeBytes. A
non-positive or non-finite maxSizeBytes is rejected at construction.
The limit is scoped per store, not per directory: each CachedStore keeps
its entries under cacheDir/<hash(storeId)> and evicts only there. Several
stores sharing one cacheDir can therefore use up to N × maxSizeBytes in
total. For stores without a derivable identity (anything other than S3/HTTP,
e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache
subdirectory is created on every process start and stale ones are never
evicted.
Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is
ever evicted — every chunk fetched from the inner store is written to
cacheDir and stays there, so sustained reads over a large dataset will
eventually fill the disk (or the pod's ephemeral-storage limit, evicting the
pod). Constructing a CachedStore without maxSizeBytes logs a
console.warn for this reason; only omit it when the working set is known to
fit on disk.
Sizing guidance:
  • Size for the hot working set, not the whole dataset — e.g. the chunks
covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and
reads fetch chunks concurrently (default concurrency 50), so usage can
transiently exceed maxSizeBytes by roughly the read concurrency × chunk
size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably
below the container's ephemeral-storage limit (or mount a dedicated volume
for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched
repeatedly); if the hit rate is low, grow the limit or narrow the access
pattern.
Shared metadata cache
open/openGroup/openArray accept a metadataCache implementing the async
Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup,
.zattrs) are served read-through: first open fetches from the store and
caches; later opens — in the same process or, with Redis, on any pod — skip
the store entirely. Entries are cached without TTL (datasets are immutable
per path). A cache error or unavailable backend falls back to the store, so
reads never fail because of the cache.
In-process:
import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });
Shared across pods via Redis (requires the optional ioredis peer
dependency — npm install ioredis):
import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });
RedisCache also accepts a connection URL directly
(new RedisCache("redis://..."), with optional ioredis options as a second
argument); the client is then created lazily on first use. Passing a
pre-configured client is preferred — with a bare URL, ioredis defaults apply
and commands issued while Redis is unreachable can stall before the store
fallback kicks in.
Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is
derived automatically for S3Store and HTTPStore; for any other store you
must pass an explicit storeId, otherwise open throws immediately
(preventing silent per-pod key divergence):
awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});
Observability hooks
Every layer accepts an optional per-instance observability object — no
global registry. The same object can be passed to multiple layers; each layer
fires only the events it owns:
import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });
A throwing (or rejecting) handler is swallowed and never breaks a read. When
no hooks are registered there is zero overhead — payload objects are not even
allocated.
Offloading decompression (worker threads)
Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks
the event loop for the whole decode — degrading the latency of every other
request in a shared API pod. gzip/zlib already run on the libuv threadpool
and are unaffected.
Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is
offloadable (currently Blosc) and whose compressed size is at least minBytes
are decoded on a worker thread; everything else decodes inline as before. Create
one pool per process, reuse it across reads, and call terminate() on shutdown
(idle workers keep the process alive).
import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();
The threshold is on the compressed size (known before decode). Use
onChunkDecoded (above) to measure decodeMs with and without the pool and
calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts
runs that A/B and also reports event-loop lag.
Reference filesystem (kerchunk)
import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();
Spatial lookups (GridIndex)
@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j)
on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it
is loaded once and queried many times — each query is pure CPU.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);
For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first
pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:
import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });
Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a
corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs
can't be trusted.
Polygon reads (readPolygon)
readPolygon streams — one time step at a time — only the cells geometrically
inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a
bounding-box block, so each backing chunk is fetched/decompressed at most once
(chunks typically span the full time axis and are reused across steps via a shared
MemoryCache), and peak memory stays bounded to ~one time slice regardless of the
time extent. Aggregation is the caller's concern — you get the raw in-polygon values.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}
resolvePolygonCells(arr, opts) returns the time-invariant selection
(cells + bbox + stride) without reading values — cells[k] aligns with
step.values[k]. Three coordinate layouts are supported:
{ kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid }
(curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured
points). Set maxCells to cap huge selections with a clamped uniform stride
(reported as selection.stride; no default cap). A runnable example lives in
examples/read-polygon.ts.
Requirements
  • Node.js >= 22
  • ESM only ("type": "module")
API
Top-level functions
FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)
All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }.
metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a
content-versioned storeId so obsolete versions' keys expire from a shared cache
instead of accumulating forever (omit ⇒ no expiry).
Store backends
ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest
Caching
ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)
Data classes
ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads
Spatial
ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)
Contributing
See CONTRIBUTING.md.
License
MIT
About
Read-only Zarr v2 array reader for Node.js
Resources
Readme
MIT license
Code of conduct
Code of conduct
Contributing
Contributing
Security policy
Security policy
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
zarr-node
CILicense: MIT
Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.
Features
  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion
Install
Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):
@i4sea:registry=https://npm.pkg.github.com

Then install:
npm install @i4sea/zarr-node
For S3 support, install the peer dependency:
npm install @aws-sdk/client-s3
Quick Start
Read an array from the filesystem
import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);
Integer dtypes (int64 / uint64)
Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array
or BigUint64Array. Their elements are bigint, not number. Coerce with
Number(value) when you need a plain number — this is safe for epoch-seconds
up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those
ranges precision is lost.
consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number
Read from HTTP
import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();
Read from S3
import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();
Connection pooling and prewarming
S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:
conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});
maxSockets (default 128, keep-alive on) caps how many chunk fetches run in
parallel — raise the read concurrency and keep maxSockets >= concurrency so a
many-chunk read finishes in one wave instead of several. Run the reader in the
same AWS region as the bucket — that, not the library, dominates latency.
Groups and multi-array reads
import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);
Bounding memory
Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk
count. By default a single get() holds at most maxInFlightBytes (256 MiB) of
decoded chunk data at once and copies each chunk into the output as it arrives,
so peak memory stays predictable even on arrays with large chunks.
// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});
Compressed point-slices pay full-chunk cost. Selecting a single
(lat, lon) from a blosc/gzip/zlib array still downloads and
decompresses the entire chunk covering that point — partial decode isn't
possible for these codecs. The cost is per chunk, not per element, so a wide
selection over a chunked axis decodes one full chunk per step. maxInFlightBytes
bounds how many of those decode concurrently; a MemoryCache avoids
re-decoding chunks across repeated reads.
readMultiple shares one budget across all arrays, so reading many
compressed arrays at once stays bounded by a single ceiling rather than
arrays × concurrency × chunkSize.
Any read whose materialized output would exceed largeReadWarningBytes
(512 MiB) — whether a full-array get() or a large slice — logs a one-line
console.warn. Set it to Infinity to silence.
Sizing maxInFlightBytes from a RAM limit
Peak bytes a single in-flight chunk holds while being processed:
peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size
per in-flight chunk; a compressed little-endian array holds 2×.
To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the
process baseline and keep a safety margin:
maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and
runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB —
remembering the read output buffer is allocated on top of the in-flight
budget. maxInFlightBytes caps the combined decoded footprint regardless of
concurrency or chunk size, so it is the binding knob for memory safety.
Caching
import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});
Eviction and cache sizing
After each write, CachedStore evicts the oldest entries by file modification
time (least-recently-written — reads do not refresh an entry's eviction
priority) so that store's cache stays at or below maxSizeBytes. A
non-positive or non-finite maxSizeBytes is rejected at construction.
The limit is scoped per store, not per directory: each CachedStore keeps
its entries under cacheDir/<hash(storeId)> and evicts only there. Several
stores sharing one cacheDir can therefore use up to N × maxSizeBytes in
total. For stores without a derivable identity (anything other than S3/HTTP,
e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache
subdirectory is created on every process start and stale ones are never
evicted.
Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is
ever evicted — every chunk fetched from the inner store is written to
cacheDir and stays there, so sustained reads over a large dataset will
eventually fill the disk (or the pod's ephemeral-storage limit, evicting the
pod). Constructing a CachedStore without maxSizeBytes logs a
console.warn for this reason; only omit it when the working set is known to
fit on disk.
Sizing guidance:
  • Size for the hot working set, not the whole dataset — e.g. the chunks
covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and
reads fetch chunks concurrently (default concurrency 50), so usage can
transiently exceed maxSizeBytes by roughly the read concurrency × chunk
size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably
below the container's ephemeral-storage limit (or mount a dedicated volume
for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched
repeatedly); if the hit rate is low, grow the limit or narrow the access
pattern.
Shared metadata cache
open/openGroup/openArray accept a metadataCache implementing the async
Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup,
.zattrs) are served read-through: first open fetches from the store and
caches; later opens — in the same process or, with Redis, on any pod — skip
the store entirely. Entries are cached without TTL (datasets are immutable
per path). A cache error or unavailable backend falls back to the store, so
reads never fail because of the cache.
In-process:
import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });
Shared across pods via Redis (requires the optional ioredis peer
dependency — npm install ioredis):
import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });
RedisCache also accepts a connection URL directly
(new RedisCache("redis://..."), with optional ioredis options as a second
argument); the client is then created lazily on first use. Passing a
pre-configured client is preferred — with a bare URL, ioredis defaults apply
and commands issued while Redis is unreachable can stall before the store
fallback kicks in.
Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is
derived automatically for S3Store and HTTPStore; for any other store you
must pass an explicit storeId, otherwise open throws immediately
(preventing silent per-pod key divergence):
awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});
Observability hooks
Every layer accepts an optional per-instance observability object — no
global registry. The same object can be passed to multiple layers; each layer
fires only the events it owns:
import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });
A throwing (or rejecting) handler is swallowed and never breaks a read. When
no hooks are registered there is zero overhead — payload objects are not even
allocated.
Offloading decompression (worker threads)
Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks
the event loop for the whole decode — degrading the latency of every other
request in a shared API pod. gzip/zlib already run on the libuv threadpool
and are unaffected.
Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is
offloadable (currently Blosc) and whose compressed size is at least minBytes
are decoded on a worker thread; everything else decodes inline as before. Create
one pool per process, reuse it across reads, and call terminate() on shutdown
(idle workers keep the process alive).
import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();
The threshold is on the compressed size (known before decode). Use
onChunkDecoded (above) to measure decodeMs with and without the pool and
calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts
runs that A/B and also reports event-loop lag.
Reference filesystem (kerchunk)
import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();
Spatial lookups (GridIndex)
@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j)
on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it
is loaded once and queried many times — each query is pure CPU.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);
For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first
pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:
import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });
Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a
corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs
can't be trusted.
Polygon reads (readPolygon)
readPolygon streams — one time step at a time — only the cells geometrically
inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a
bounding-box block, so each backing chunk is fetched/decompressed at most once
(chunks typically span the full time axis and are reused across steps via a shared
MemoryCache), and peak memory stays bounded to ~one time slice regardless of the
time extent. Aggregation is the caller's concern — you get the raw in-polygon values.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}
resolvePolygonCells(arr, opts) returns the time-invariant selection
(cells + bbox + stride) without reading values — cells[k] aligns with
step.values[k]. Three coordinate layouts are supported:
{ kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid }
(curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured
points). Set maxCells to cap huge selections with a clamped uniform stride
(reported as selection.stride; no default cap). A runnable example lives in
examples/read-polygon.ts.
Requirements
  • Node.js >= 22
  • ESM only ("type": "module")
API
Top-level functions
FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)
All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }.
metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a
content-versioned storeId so obsolete versions' keys expire from a shared cache
instead of accumulating forever (omit ⇒ no expiry).
Store backends
ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest
Caching
ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)
Data classes
ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads
Spatial
ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)
Contributing
See CONTRIBUTING.md.
License
MIT
About
Read-only Zarr v2 array reader for Node.js
Resources
Readme
MIT license
Code of conduct
Code of conduct
Contributing
Contributing
Security policy
Security policy
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
zarr-node
CILicense: MIT
Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.
Features
  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion
Install
Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):
@i4sea:registry=https://npm.pkg.github.com

Then install:
npm install @i4sea/zarr-node
For S3 support, install the peer dependency:
npm install @aws-sdk/client-s3
Quick Start
Read an array from the filesystem
import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);
Integer dtypes (int64 / uint64)
Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array
or BigUint64Array. Their elements are bigint, not number. Coerce with
Number(value) when you need a plain number — this is safe for epoch-seconds
up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those
ranges precision is lost.
consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number
Read from HTTP
import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();
Read from S3
import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();
Connection pooling and prewarming
S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:
conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});
maxSockets (default 128, keep-alive on) caps how many chunk fetches run in
parallel — raise the read concurrency and keep maxSockets >= concurrency so a
many-chunk read finishes in one wave instead of several. Run the reader in the
same AWS region as the bucket — that, not the library, dominates latency.
Groups and multi-array reads
import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);
Bounding memory
Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk
count. By default a single get() holds at most maxInFlightBytes (256 MiB) of
decoded chunk data at once and copies each chunk into the output as it arrives,
so peak memory stays predictable even on arrays with large chunks.
// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});
Compressed point-slices pay full-chunk cost. Selecting a single
(lat, lon) from a blosc/gzip/zlib array still downloads and
decompresses the entire chunk covering that point — partial decode isn't
possible for these codecs. The cost is per chunk, not per element, so a wide
selection over a chunked axis decodes one full chunk per step. maxInFlightBytes
bounds how many of those decode concurrently; a MemoryCache avoids
re-decoding chunks across repeated reads.
readMultiple shares one budget across all arrays, so reading many
compressed arrays at once stays bounded by a single ceiling rather than
arrays × concurrency × chunkSize.
Any read whose materialized output would exceed largeReadWarningBytes
(512 MiB) — whether a full-array get() or a large slice — logs a one-line
console.warn. Set it to Infinity to silence.
Sizing maxInFlightBytes from a RAM limit
Peak bytes a single in-flight chunk holds while being processed:
peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size
per in-flight chunk; a compressed little-endian array holds 2×.
To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the
process baseline and keep a safety margin:
maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and
runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB —
remembering the read output buffer is allocated on top of the in-flight
budget. maxInFlightBytes caps the combined decoded footprint regardless of
concurrency or chunk size, so it is the binding knob for memory safety.
Caching
import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});
Eviction and cache sizing
After each write, CachedStore evicts the oldest entries by file modification
time (least-recently-written — reads do not refresh an entry's eviction
priority) so that store's cache stays at or below maxSizeBytes. A
non-positive or non-finite maxSizeBytes is rejected at construction.
The limit is scoped per store, not per directory: each CachedStore keeps
its entries under cacheDir/<hash(storeId)> and evicts only there. Several
stores sharing one cacheDir can therefore use up to N × maxSizeBytes in
total. For stores without a derivable identity (anything other than S3/HTTP,
e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache
subdirectory is created on every process start and stale ones are never
evicted.
Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is
ever evicted — every chunk fetched from the inner store is written to
cacheDir and stays there, so sustained reads over a large dataset will
eventually fill the disk (or the pod's ephemeral-storage limit, evicting the
pod). Constructing a CachedStore without maxSizeBytes logs a
console.warn for this reason; only omit it when the working set is known to
fit on disk.
Sizing guidance:
  • Size for the hot working set, not the whole dataset — e.g. the chunks
covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and
reads fetch chunks concurrently (default concurrency 50), so usage can
transiently exceed maxSizeBytes by roughly the read concurrency × chunk
size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably
below the container's ephemeral-storage limit (or mount a dedicated volume
for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched
repeatedly); if the hit rate is low, grow the limit or narrow the access
pattern.
Shared metadata cache
open/openGroup/openArray accept a metadataCache implementing the async
Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup,
.zattrs) are served read-through: first open fetches from the store and
caches; later opens — in the same process or, with Redis, on any pod — skip
the store entirely. Entries are cached without TTL (datasets are immutable
per path). A cache error or unavailable backend falls back to the store, so
reads never fail because of the cache.
In-process:
import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });
Shared across pods via Redis (requires the optional ioredis peer
dependency — npm install ioredis):
import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });
RedisCache also accepts a connection URL directly
(new RedisCache("redis://..."), with optional ioredis options as a second
argument); the client is then created lazily on first use. Passing a
pre-configured client is preferred — with a bare URL, ioredis defaults apply
and commands issued while Redis is unreachable can stall before the store
fallback kicks in.
Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is
derived automatically for S3Store and HTTPStore; for any other store you
must pass an explicit storeId, otherwise open throws immediately
(preventing silent per-pod key divergence):
awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});
Observability hooks
Every layer accepts an optional per-instance observability object — no
global registry. The same object can be passed to multiple layers; each layer
fires only the events it owns:
import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });
A throwing (or rejecting) handler is swallowed and never breaks a read. When
no hooks are registered there is zero overhead — payload objects are not even
allocated.
Offloading decompression (worker threads)
Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks
the event loop for the whole decode — degrading the latency of every other
request in a shared API pod. gzip/zlib already run on the libuv threadpool
and are unaffected.
Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is
offloadable (currently Blosc) and whose compressed size is at least minBytes
are decoded on a worker thread; everything else decodes inline as before. Create
one pool per process, reuse it across reads, and call terminate() on shutdown
(idle workers keep the process alive).
import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();
The threshold is on the compressed size (known before decode). Use
onChunkDecoded (above) to measure decodeMs with and without the pool and
calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts
runs that A/B and also reports event-loop lag.
Reference filesystem (kerchunk)
import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();
Spatial lookups (GridIndex)
@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j)
on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it
is loaded once and queried many times — each query is pure CPU.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);
For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first
pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:
import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });
Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a
corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs
can't be trusted.
Polygon reads (readPolygon)
readPolygon streams — one time step at a time — only the cells geometrically
inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a
bounding-box block, so each backing chunk is fetched/decompressed at most once
(chunks typically span the full time axis and are reused across steps via a shared
MemoryCache), and peak memory stays bounded to ~one time slice regardless of the
time extent. Aggregation is the caller's concern — you get the raw in-polygon values.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}
resolvePolygonCells(arr, opts) returns the time-invariant selection
(cells + bbox + stride) without reading values — cells[k] aligns with
step.values[k]. Three coordinate layouts are supported:
{ kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid }
(curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured
points). Set maxCells to cap huge selections with a clamped uniform stride
(reported as selection.stride; no default cap). A runnable example lives in
examples/read-polygon.ts.
Requirements
  • Node.js >= 22
  • ESM only ("type": "module")
API
Top-level functions
FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)
All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }.
metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a
content-versioned storeId so obsolete versions' keys expire from a shared cache
instead of accumulating forever (omit ⇒ no expiry).
Store backends
ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest
Caching
ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)
Data classes
ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads
Spatial
ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)
Contributing
See CONTRIBUTING.md.
License
MIT
About
Read-only Zarr v2 array reader for Node.js
Resources
Readme
MIT license
Code of conduct
Code of conduct
Contributing
Contributing
Security policy
Security policy
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
zarr-node
CILicense: MIT
Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.
Features
  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion
Install
Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):
@i4sea:registry=https://npm.pkg.github.com

Then install:
npm install @i4sea/zarr-node
For S3 support, install the peer dependency:
npm install @aws-sdk/client-s3
Quick Start
Read an array from the filesystem
import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);
Integer dtypes (int64 / uint64)
Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array
or BigUint64Array. Their elements are bigint, not number. Coerce with
Number(value) when you need a plain number — this is safe for epoch-seconds
up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those
ranges precision is lost.
consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number
Read from HTTP
import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();
Read from S3
import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();
Connection pooling and prewarming
S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:
conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});
maxSockets (default 128, keep-alive on) caps how many chunk fetches run in
parallel — raise the read concurrency and keep maxSockets >= concurrency so a
many-chunk read finishes in one wave instead of several. Run the reader in the
same AWS region as the bucket — that, not the library, dominates latency.
Groups and multi-array reads
import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);
Bounding memory
Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk
count. By default a single get() holds at most maxInFlightBytes (256 MiB) of
decoded chunk data at once and copies each chunk into the output as it arrives,
so peak memory stays predictable even on arrays with large chunks.
// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});
Compressed point-slices pay full-chunk cost. Selecting a single
(lat, lon) from a blosc/gzip/zlib array still downloads and
decompresses the entire chunk covering that point — partial decode isn't
possible for these codecs. The cost is per chunk, not per element, so a wide
selection over a chunked axis decodes one full chunk per step. maxInFlightBytes
bounds how many of those decode concurrently; a MemoryCache avoids
re-decoding chunks across repeated reads.
readMultiple shares one budget across all arrays, so reading many
compressed arrays at once stays bounded by a single ceiling rather than
arrays × concurrency × chunkSize.
Any read whose materialized output would exceed largeReadWarningBytes
(512 MiB) — whether a full-array get() or a large slice — logs a one-line
console.warn. Set it to Infinity to silence.
Sizing maxInFlightBytes from a RAM limit
Peak bytes a single in-flight chunk holds while being processed:
peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size
per in-flight chunk; a compressed little-endian array holds 2×.
To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the
process baseline and keep a safety margin:
maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and
runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB —
remembering the read output buffer is allocated on top of the in-flight
budget. maxInFlightBytes caps the combined decoded footprint regardless of
concurrency or chunk size, so it is the binding knob for memory safety.
Caching
import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});
Eviction and cache sizing
After each write, CachedStore evicts the oldest entries by file modification
time (least-recently-written — reads do not refresh an entry's eviction
priority) so that store's cache stays at or below maxSizeBytes. A
non-positive or non-finite maxSizeBytes is rejected at construction.
The limit is scoped per store, not per directory: each CachedStore keeps
its entries under cacheDir/<hash(storeId)> and evicts only there. Several
stores sharing one cacheDir can therefore use up to N × maxSizeBytes in
total. For stores without a derivable identity (anything other than S3/HTTP,
e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache
subdirectory is created on every process start and stale ones are never
evicted.
Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is
ever evicted — every chunk fetched from the inner store is written to
cacheDir and stays there, so sustained reads over a large dataset will
eventually fill the disk (or the pod's ephemeral-storage limit, evicting the
pod). Constructing a CachedStore without maxSizeBytes logs a
console.warn for this reason; only omit it when the working set is known to
fit on disk.
Sizing guidance:
  • Size for the hot working set, not the whole dataset — e.g. the chunks
covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and
reads fetch chunks concurrently (default concurrency 50), so usage can
transiently exceed maxSizeBytes by roughly the read concurrency × chunk
size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably
below the container's ephemeral-storage limit (or mount a dedicated volume
for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched
repeatedly); if the hit rate is low, grow the limit or narrow the access
pattern.
Shared metadata cache
open/openGroup/openArray accept a metadataCache implementing the async
Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup,
.zattrs) are served read-through: first open fetches from the store and
caches; later opens — in the same process or, with Redis, on any pod — skip
the store entirely. Entries are cached without TTL (datasets are immutable
per path). A cache error or unavailable backend falls back to the store, so
reads never fail because of the cache.
In-process:
import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });
Shared across pods via Redis (requires the optional ioredis peer
dependency — npm install ioredis):
import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });
RedisCache also accepts a connection URL directly
(new RedisCache("redis://..."), with optional ioredis options as a second
argument); the client is then created lazily on first use. Passing a
pre-configured client is preferred — with a bare URL, ioredis defaults apply
and commands issued while Redis is unreachable can stall before the store
fallback kicks in.
Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is
derived automatically for S3Store and HTTPStore; for any other store you
must pass an explicit storeId, otherwise open throws immediately
(preventing silent per-pod key divergence):
awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});
Observability hooks
Every layer accepts an optional per-instance observability object — no
global registry. The same object can be passed to multiple layers; each layer
fires only the events it owns:
import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });
A throwing (or rejecting) handler is swallowed and never breaks a read. When
no hooks are registered there is zero overhead — payload objects are not even
allocated.
Offloading decompression (worker threads)
Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks
the event loop for the whole decode — degrading the latency of every other
request in a shared API pod. gzip/zlib already run on the libuv threadpool
and are unaffected.
Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is
offloadable (currently Blosc) and whose compressed size is at least minBytes
are decoded on a worker thread; everything else decodes inline as before. Create
one pool per process, reuse it across reads, and call terminate() on shutdown
(idle workers keep the process alive).
import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();
The threshold is on the compressed size (known before decode). Use
onChunkDecoded (above) to measure decodeMs with and without the pool and
calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts
runs that A/B and also reports event-loop lag.
Reference filesystem (kerchunk)
import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();
Spatial lookups (GridIndex)
@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j)
on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it
is loaded once and queried many times — each query is pure CPU.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);
For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first
pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:
import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });
Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a
corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs
can't be trusted.
Polygon reads (readPolygon)
readPolygon streams — one time step at a time — only the cells geometrically
inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a
bounding-box block, so each backing chunk is fetched/decompressed at most once
(chunks typically span the full time axis and are reused across steps via a shared
MemoryCache), and peak memory stays bounded to ~one time slice regardless of the
time extent. Aggregation is the caller's concern — you get the raw in-polygon values.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}
resolvePolygonCells(arr, opts) returns the time-invariant selection
(cells + bbox + stride) without reading values — cells[k] aligns with
step.values[k]. Three coordinate layouts are supported:
{ kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid }
(curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured
points). Set maxCells to cap huge selections with a clamped uniform stride
(reported as selection.stride; no default cap). A runnable example lives in
examples/read-polygon.ts.
Requirements
  • Node.js >= 22
  • ESM only ("type": "module")
API
Top-level functions
FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)
All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }.
metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a
content-versioned storeId so obsolete versions' keys expire from a shared cache
instead of accumulating forever (omit ⇒ no expiry).
Store backends
ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest
Caching
ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)
Data classes
ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads
Spatial
ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)
Contributing
See CONTRIBUTING.md.
License
MIT
About
Read-only Zarr v2 array reader for Node.js
Resources
Readme
MIT license
Code of conduct
Code of conduct
Contributing
Contributing
Security policy
Security policy
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
zarr-node
CILicense: MIT
Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.
Features
  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion
Install
Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):
@i4sea:registry=https://npm.pkg.github.com

Then install:
npm install @i4sea/zarr-node
For S3 support, install the peer dependency:
npm install @aws-sdk/client-s3
Quick Start
Read an array from the filesystem
import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);
Integer dtypes (int64 / uint64)
Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array
or BigUint64Array. Their elements are bigint, not number. Coerce with
Number(value) when you need a plain number — this is safe for epoch-seconds
up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those
ranges precision is lost.
consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number
Read from HTTP
import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();
Read from S3
import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();
Connection pooling and prewarming
S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:
conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});
maxSockets (default 128, keep-alive on) caps how many chunk fetches run in
parallel — raise the read concurrency and keep maxSockets >= concurrency so a
many-chunk read finishes in one wave instead of several. Run the reader in the
same AWS region as the bucket — that, not the library, dominates latency.
Groups and multi-array reads
import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);
Bounding memory
Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk
count. By default a single get() holds at most maxInFlightBytes (256 MiB) of
decoded chunk data at once and copies each chunk into the output as it arrives,
so peak memory stays predictable even on arrays with large chunks.
// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});
Compressed point-slices pay full-chunk cost. Selecting a single
(lat, lon) from a blosc/gzip/zlib array still downloads and
decompresses the entire chunk covering that point — partial decode isn't
possible for these codecs. The cost is per chunk, not per element, so a wide
selection over a chunked axis decodes one full chunk per step. maxInFlightBytes
bounds how many of those decode concurrently; a MemoryCache avoids
re-decoding chunks across repeated reads.
readMultiple shares one budget across all arrays, so reading many
compressed arrays at once stays bounded by a single ceiling rather than
arrays × concurrency × chunkSize.
Any read whose materialized output would exceed largeReadWarningBytes
(512 MiB) — whether a full-array get() or a large slice — logs a one-line
console.warn. Set it to Infinity to silence.
Sizing maxInFlightBytes from a RAM limit
Peak bytes a single in-flight chunk holds while being processed:
peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size
per in-flight chunk; a compressed little-endian array holds 2×.
To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the
process baseline and keep a safety margin:
maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and
runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB —
remembering the read output buffer is allocated on top of the in-flight
budget. maxInFlightBytes caps the combined decoded footprint regardless of
concurrency or chunk size, so it is the binding knob for memory safety.
Caching
import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});
Eviction and cache sizing
After each write, CachedStore evicts the oldest entries by file modification
time (least-recently-written — reads do not refresh an entry's eviction
priority) so that store's cache stays at or below maxSizeBytes. A
non-positive or non-finite maxSizeBytes is rejected at construction.
The limit is scoped per store, not per directory: each CachedStore keeps
its entries under cacheDir/<hash(storeId)> and evicts only there. Several
stores sharing one cacheDir can therefore use up to N × maxSizeBytes in
total. For stores without a derivable identity (anything other than S3/HTTP,
e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache
subdirectory is created on every process start and stale ones are never
evicted.
Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is
ever evicted — every chunk fetched from the inner store is written to
cacheDir and stays there, so sustained reads over a large dataset will
eventually fill the disk (or the pod's ephemeral-storage limit, evicting the
pod). Constructing a CachedStore without maxSizeBytes logs a
console.warn for this reason; only omit it when the working set is known to
fit on disk.
Sizing guidance:
  • Size for the hot working set, not the whole dataset — e.g. the chunks
covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and
reads fetch chunks concurrently (default concurrency 50), so usage can
transiently exceed maxSizeBytes by roughly the read concurrency × chunk
size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably
below the container's ephemeral-storage limit (or mount a dedicated volume
for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched
repeatedly); if the hit rate is low, grow the limit or narrow the access
pattern.
Shared metadata cache
open/openGroup/openArray accept a metadataCache implementing the async
Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup,
.zattrs) are served read-through: first open fetches from the store and
caches; later opens — in the same process or, with Redis, on any pod — skip
the store entirely. Entries are cached without TTL (datasets are immutable
per path). A cache error or unavailable backend falls back to the store, so
reads never fail because of the cache.
In-process:
import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });
Shared across pods via Redis (requires the optional ioredis peer
dependency — npm install ioredis):
import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });
RedisCache also accepts a connection URL directly
(new RedisCache("redis://..."), with optional ioredis options as a second
argument); the client is then created lazily on first use. Passing a
pre-configured client is preferred — with a bare URL, ioredis defaults apply
and commands issued while Redis is unreachable can stall before the store
fallback kicks in.
Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is
derived automatically for S3Store and HTTPStore; for any other store you
must pass an explicit storeId, otherwise open throws immediately
(preventing silent per-pod key divergence):
awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});
Observability hooks
Every layer accepts an optional per-instance observability object — no
global registry. The same object can be passed to multiple layers; each layer
fires only the events it owns:
import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });
A throwing (or rejecting) handler is swallowed and never breaks a read. When
no hooks are registered there is zero overhead — payload objects are not even
allocated.
Offloading decompression (worker threads)
Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks
the event loop for the whole decode — degrading the latency of every other
request in a shared API pod. gzip/zlib already run on the libuv threadpool
and are unaffected.
Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is
offloadable (currently Blosc) and whose compressed size is at least minBytes
are decoded on a worker thread; everything else decodes inline as before. Create
one pool per process, reuse it across reads, and call terminate() on shutdown
(idle workers keep the process alive).
import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();
The threshold is on the compressed size (known before decode). Use
onChunkDecoded (above) to measure decodeMs with and without the pool and
calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts
runs that A/B and also reports event-loop lag.
Reference filesystem (kerchunk)
import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();
Spatial lookups (GridIndex)
@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j)
on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it
is loaded once and queried many times — each query is pure CPU.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);
For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first
pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:
import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });
Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a
corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs
can't be trusted.
Polygon reads (readPolygon)
readPolygon streams — one time step at a time — only the cells geometrically
inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a
bounding-box block, so each backing chunk is fetched/decompressed at most once
(chunks typically span the full time axis and are reused across steps via a shared
MemoryCache), and peak memory stays bounded to ~one time slice regardless of the
time extent. Aggregation is the caller's concern — you get the raw in-polygon values.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}
resolvePolygonCells(arr, opts) returns the time-invariant selection
(cells + bbox + stride) without reading values — cells[k] aligns with
step.values[k]. Three coordinate layouts are supported:
{ kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid }
(curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured
points). Set maxCells to cap huge selections with a clamped uniform stride
(reported as selection.stride; no default cap). A runnable example lives in
examples/read-polygon.ts.
Requirements
  • Node.js >= 22
  • ESM only ("type": "module")
API
Top-level functions
FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)
All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }.
metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a
content-versioned storeId so obsolete versions' keys expire from a shared cache
instead of accumulating forever (omit ⇒ no expiry).
Store backends
ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest
Caching
ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)
Data classes
ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads
Spatial
ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)
Contributing
See CONTRIBUTING.md.
License
MIT
About
Read-only Zarr v2 array reader for Node.js
Resources
Readme
MIT license
Code of conduct
Code of conduct
Contributing
Contributing
Security policy
Security policy
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
zarr-node
CILicense: MIT
Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.
Features
  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion
Install
Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):
@i4sea:registry=https://npm.pkg.github.com

Then install:
npm install @i4sea/zarr-node
For S3 support, install the peer dependency:
npm install @aws-sdk/client-s3
Quick Start
Read an array from the filesystem
import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);
Integer dtypes (int64 / uint64)
Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array
or BigUint64Array. Their elements are bigint, not number. Coerce with
Number(value) when you need a plain number — this is safe for epoch-seconds
up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those
ranges precision is lost.
consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number
Read from HTTP
import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();
Read from S3
import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();
Connection pooling and prewarming
S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:
conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});
maxSockets (default 128, keep-alive on) caps how many chunk fetches run in
parallel — raise the read concurrency and keep maxSockets >= concurrency so a
many-chunk read finishes in one wave instead of several. Run the reader in the
same AWS region as the bucket — that, not the library, dominates latency.
Groups and multi-array reads
import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);
Bounding memory
Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk
count. By default a single get() holds at most maxInFlightBytes (256 MiB) of
decoded chunk data at once and copies each chunk into the output as it arrives,
so peak memory stays predictable even on arrays with large chunks.
// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});
Compressed point-slices pay full-chunk cost. Selecting a single
(lat, lon) from a blosc/gzip/zlib array still downloads and
decompresses the entire chunk covering that point — partial decode isn't
possible for these codecs. The cost is per chunk, not per element, so a wide
selection over a chunked axis decodes one full chunk per step. maxInFlightBytes
bounds how many of those decode concurrently; a MemoryCache avoids
re-decoding chunks across repeated reads.
readMultiple shares one budget across all arrays, so reading many
compressed arrays at once stays bounded by a single ceiling rather than
arrays × concurrency × chunkSize.
Any read whose materialized output would exceed largeReadWarningBytes
(512 MiB) — whether a full-array get() or a large slice — logs a one-line
console.warn. Set it to Infinity to silence.
Sizing maxInFlightBytes from a RAM limit
Peak bytes a single in-flight chunk holds while being processed:
peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size
per in-flight chunk; a compressed little-endian array holds 2×.
To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the
process baseline and keep a safety margin:
maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and
runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB —
remembering the read output buffer is allocated on top of the in-flight
budget. maxInFlightBytes caps the combined decoded footprint regardless of
concurrency or chunk size, so it is the binding knob for memory safety.
Caching
import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});
Eviction and cache sizing
After each write, CachedStore evicts the oldest entries by file modification
time (least-recently-written — reads do not refresh an entry's eviction
priority) so that store's cache stays at or below maxSizeBytes. A
non-positive or non-finite maxSizeBytes is rejected at construction.
The limit is scoped per store, not per directory: each CachedStore keeps
its entries under cacheDir/<hash(storeId)> and evicts only there. Several
stores sharing one cacheDir can therefore use up to N × maxSizeBytes in
total. For stores without a derivable identity (anything other than S3/HTTP,
e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache
subdirectory is created on every process start and stale ones are never
evicted.
Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is
ever evicted — every chunk fetched from the inner store is written to
cacheDir and stays there, so sustained reads over a large dataset will
eventually fill the disk (or the pod's ephemeral-storage limit, evicting the
pod). Constructing a CachedStore without maxSizeBytes logs a
console.warn for this reason; only omit it when the working set is known to
fit on disk.
Sizing guidance:
  • Size for the hot working set, not the whole dataset — e.g. the chunks
covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and
reads fetch chunks concurrently (default concurrency 50), so usage can
transiently exceed maxSizeBytes by roughly the read concurrency × chunk
size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably
below the container's ephemeral-storage limit (or mount a dedicated volume
for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched
repeatedly); if the hit rate is low, grow the limit or narrow the access
pattern.
Shared metadata cache
open/openGroup/openArray accept a metadataCache implementing the async
Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup,
.zattrs) are served read-through: first open fetches from the store and
caches; later opens — in the same process or, with Redis, on any pod — skip
the store entirely. Entries are cached without TTL (datasets are immutable
per path). A cache error or unavailable backend falls back to the store, so
reads never fail because of the cache.
In-process:
import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });
Shared across pods via Redis (requires the optional ioredis peer
dependency — npm install ioredis):
import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });
RedisCache also accepts a connection URL directly
(new RedisCache("redis://..."), with optional ioredis options as a second
argument); the client is then created lazily on first use. Passing a
pre-configured client is preferred — with a bare URL, ioredis defaults apply
and commands issued while Redis is unreachable can stall before the store
fallback kicks in.
Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is
derived automatically for S3Store and HTTPStore; for any other store you
must pass an explicit storeId, otherwise open throws immediately
(preventing silent per-pod key divergence):
awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});
Observability hooks
Every layer accepts an optional per-instance observability object — no
global registry. The same object can be passed to multiple layers; each layer
fires only the events it owns:
import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });
A throwing (or rejecting) handler is swallowed and never breaks a read. When
no hooks are registered there is zero overhead — payload objects are not even
allocated.
Offloading decompression (worker threads)
Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks
the event loop for the whole decode — degrading the latency of every other
request in a shared API pod. gzip/zlib already run on the libuv threadpool
and are unaffected.
Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is
offloadable (currently Blosc) and whose compressed size is at least minBytes
are decoded on a worker thread; everything else decodes inline as before. Create
one pool per process, reuse it across reads, and call terminate() on shutdown
(idle workers keep the process alive).
import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();
The threshold is on the compressed size (known before decode). Use
onChunkDecoded (above) to measure decodeMs with and without the pool and
calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts
runs that A/B and also reports event-loop lag.
Reference filesystem (kerchunk)
import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();
Spatial lookups (GridIndex)
@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j)
on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it
is loaded once and queried many times — each query is pure CPU.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);
For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first
pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:
import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });
Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a
corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs
can't be trusted.
Polygon reads (readPolygon)
readPolygon streams — one time step at a time — only the cells geometrically
inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a
bounding-box block, so each backing chunk is fetched/decompressed at most once
(chunks typically span the full time axis and are reused across steps via a shared
MemoryCache), and peak memory stays bounded to ~one time slice regardless of the
time extent. Aggregation is the caller's concern — you get the raw in-polygon values.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}
resolvePolygonCells(arr, opts) returns the time-invariant selection
(cells + bbox + stride) without reading values — cells[k] aligns with
step.values[k]. Three coordinate layouts are supported:
{ kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid }
(curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured
points). Set maxCells to cap huge selections with a clamped uniform stride
(reported as selection.stride; no default cap). A runnable example lives in
examples/read-polygon.ts.
Requirements
  • Node.js >= 22
  • ESM only ("type": "module")
API
Top-level functions
FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)
All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }.
metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a
content-versioned storeId so obsolete versions' keys expire from a shared cache
instead of accumulating forever (omit ⇒ no expiry).
Store backends
ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest
Caching
ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)
Data classes
ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads
Spatial
ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)
Contributing
See CONTRIBUTING.md.
License
MIT
About
Read-only Zarr v2 array reader for Node.js
Resources
Readme
MIT license
Code of conduct
Code of conduct
Contributing
Contributing
Security policy
Security policy
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
zarr-node
CILicense: MIT
Read-only Zarr v2 array reader for Node.js. Server-first, with FileSystem, HTTP, and S3 backends.
Features
  • Zarr v2 chunked array reader with full dtype support
  • Three storage backends: FileSystem, HTTP (with retry/timeout), S3
  • Consolidated metadata (.zmetadata) for fast group discovery
  • Disk cache with thundering herd protection and LRU eviction
  • In-memory LRU cache for sub-millisecond repeated reads
  • Shared metadata cache — pluggable Cache interface with in-memory and Redis adapters
  • Observability hooks — per-instance callbacks for cache hits/misses, store fetches, retries, decodes, in-flight bytes, and missing chunks
  • Built-in Blosc codec (lz4, zstd, zlib, snappy) — zero configuration
  • Byte-range requests for partial chunk fetches on uncompressed data
  • Bounded memory — reads cap decoded bytes in flight, not just chunk count
  • Multi-array reads sharing one in-flight memory budget
  • Reference filesystem (kerchunk) for reading HDF5/NetCDF without conversion
Install
Published to GitHub Packages under the @i4sea scope. Add this to a .npmrc in your consumer project (or ~/.npmrc):
@i4sea:registry=https://npm.pkg.github.com

Then install:
npm install @i4sea/zarr-node
For S3 support, install the peer dependency:
npm install @aws-sdk/client-s3
Quick Start
Read an array from the filesystem
import{FileSystemStore,open}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr"});constarray=awaitopen(store);// Read all dataconstdata=awaitarray.read();// Read a slice (first 10 rows, columns 5-15)constslice=awaitarray.read([[0,10],[5,15],]);
Integer dtypes (int64 / uint64)
Arrays with dtype <i8, >i8, <u8, or >u8 are returned as BigInt64Array
or BigUint64Array. Their elements are bigint, not number. Coerce with
Number(value) when you need a plain number — this is safe for epoch-seconds
up to year 285K AD and for epoch-nanoseconds up to year 2262. Beyond those
ranges precision is lost.
consttimeArray=awaitgroup.getArray("time");// dtype "<i8"constdata=awaittimeArray.read();// BigInt64Arrayconstseconds=Number(data[0]);// bigint -> number
Read from HTTP
import{HTTPStore,open}from"@i4sea/zarr-node";conststore=newHTTPStore({url: "https://example.com/data.zarr"});constarray=awaitopen(store);constdata=awaitarray.read();
Read from S3
import{S3Store,open}from"@i4sea/zarr-node";conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",});constarray=awaitopen(store);constdata=awaitarray.read();
Connection pooling and prewarming
S3 reads are latency-bound: each chunk is one round trip. Two levers reduce that:
conststore=newS3Store({bucket: "my-bucket",prefix: "data.zarr",region: "us-east-1",maxSockets: 256,// keep-alive pool size (default 128). Set >= read concurrency.warmOnCreate: true,// open a TLS connection up front (or call store.prewarm())});awaitstore.prewarm();// optional explicit warm-up at pod startupconstdata=awaitarray.read(undefined,{concurrency: 200});
maxSockets (default 128, keep-alive on) caps how many chunk fetches run in
parallel — raise the read concurrency and keep maxSockets >= concurrency so a
many-chunk read finishes in one wave instead of several. Run the reader in the
same AWS region as the bucket — that, not the library, dominates latency.
Groups and multi-array reads
import{FileSystemStore,openGroup}from"@i4sea/zarr-node";conststore=newFileSystemStore({path: "/path/to/zarr-group"});constgroup=awaitopenGroup(store);// List arraysconstarrays=awaitgroup.arrays();// Read multiple arrays at once (shared in-flight memory budget)constresults=awaitgroup.readMultiple(["temperature","humidity","wind"],[[0,10]],);
Bounding memory
Reads are bounded by a decoded-bytes-in-flight budget, not just a chunk
count. By default a single get() holds at most maxInFlightBytes (256 MiB) of
decoded chunk data at once and copies each chunk into the output as it arrives,
so peak memory stays predictable even on arrays with large chunks.
// Point over a full axis on a compressed array — bound the decode footprint// explicitly (otherwise the 256 MiB default applies).constseries=awaitarray.get([null,latIdx,lonIdx],{maxInFlightBytes: 64*1024*1024,// 64 MiB live at onceconcurrency: 8,// network-request cap; the byte budget binds first on big chunks});
Compressed point-slices pay full-chunk cost. Selecting a single
(lat, lon) from a blosc/gzip/zlib array still downloads and
decompresses the entire chunk covering that point — partial decode isn't
possible for these codecs. The cost is per chunk, not per element, so a wide
selection over a chunked axis decodes one full chunk per step. maxInFlightBytes
bounds how many of those decode concurrently; a MemoryCache avoids
re-decoding chunks across repeated reads.
readMultiple shares one budget across all arrays, so reading many
compressed arrays at once stays bounded by a single ceiling rather than
arrays × concurrency × chunkSize.
Any read whose materialized output would exceed largeReadWarningBytes
(512 MiB) — whether a full-array get() or a large slice — logs a one-line
console.warn. Set it to Infinity to silence.
Sizing maxInFlightBytes from a RAM limit
Peak bytes a single in-flight chunk holds while being processed:
peakPerChunk = chunkBytes × (decodeFactor + byteSwapFactor)
decodeFactor = 2 if the array is compressed (compressed input + decoded
output coexist during decode), else 1
byteSwapFactor = 1 if the dtype is big-endian (an extra copy is made before
the in-place byte swap), else 0

So a compressed, big-endian array transiently holds up to 3× its chunk size
per in-flight chunk; a compressed little-endian array holds 2×.
To derive a safe maxInFlightBytes from a pod's RAM limit, subtract the
process baseline and keep a safety margin:
maxInFlightBytes ≈ (podRamLimit − baselineHeap) × safetyFraction

For example, a pod with a 2 GiB memory limit, ~300 MiB of baseline heap and
runtime, and a 0.5 safety fraction supports maxInFlightBytes ≈ 850 MiB —
remembering the read output buffer is allocated on top of the in-flight
budget. maxInFlightBytes caps the combined decoded footprint regardless of
concurrency or chunk size, so it is the binding knob for memory safety.
Caching
import{FileSystemStore,CachedStore,MemoryCache,open}from"@i4sea/zarr-node";// Disk cache (persists across restarts)constinner=newFileSystemStore({path: "/path/to/zarr"});conststore=newCachedStore(inner,{cacheDir: "/tmp/zarr-cache",storeId: "my-dataset",// stable cache identity across restartsmaxSizeBytes: 500*1024*1024,// 500 MB limit});// In-memory cache (for hot data)constmemCache=newMemoryCache({maxBytes: 100*1024*1024});// 100 MBconstarray=awaitopen(store);constdata=awaitarray.read(undefined,{memoryCache: memCache});
Eviction and cache sizing
After each write, CachedStore evicts the oldest entries by file modification
time (least-recently-written — reads do not refresh an entry's eviction
priority) so that store's cache stays at or below maxSizeBytes. A
non-positive or non-finite maxSizeBytes is rejected at construction.
The limit is scoped per store, not per directory: each CachedStore keeps
its entries under cacheDir/<hash(storeId)> and evicts only there. Several
stores sharing one cacheDir can therefore use up to N × maxSizeBytes in
total. For stores without a derivable identity (anything other than S3/HTTP,
e.g. FileSystemStore), pass an explicit storeId — otherwise a new cache
subdirectory is created on every process start and stale ones are never
evicted.
Unbounded-growth risk: maxSizeBytes is optional. Without it, nothing is
ever evicted — every chunk fetched from the inner store is written to
cacheDir and stays there, so sustained reads over a large dataset will
eventually fill the disk (or the pod's ephemeral-storage limit, evicting the
pod). Constructing a CachedStore without maxSizeBytes logs a
console.warn for this reason; only omit it when the working set is known to
fit on disk.
Sizing guidance:
  • Size for the hot working set, not the whole dataset — e.g. the chunks
covering the time window and variables your queries actually touch.
  • Leave headroom on the volume: eviction runs after each chunk is written and
reads fetch chunks concurrently (default concurrency 50), so usage can
transiently exceed maxSizeBytes by roughly the read concurrency × chunk
size before settling back under the limit.
  • In Kubernetes, keep maxSizeBytes (plus the headroom above) comfortably
below the container's ephemeral-storage limit (or mount a dedicated volume
for cacheDir).
  • Too small a limit causes thrashing (chunks are evicted and re-fetched
repeatedly); if the hit rate is low, grow the limit or narrow the access
pattern.
Shared metadata cache
open/openGroup/openArray accept a metadataCache implementing the async
Cache interface. Metadata reads (.zmetadata, .zarray, .zgroup,
.zattrs) are served read-through: first open fetches from the store and
caches; later opens — in the same process or, with Redis, on any pod — skip
the store entirely. Entries are cached without TTL (datasets are immutable
per path). A cache error or unavailable backend falls back to the store, so
reads never fail because of the cache.
In-process:
import{InMemoryCache,open}from"@i4sea/zarr-node";constmetadataCache=newInMemoryCache({maxBytes: 64*1024*1024});constgroup=awaitopen(store,"",{ metadataCache });
Shared across pods via Redis (requires the optional ioredis peer
dependency — npm install ioredis):
import{open}from"@i4sea/zarr-node";import{RedisCache}from"@i4sea/zarr-node/redis";importRedisfrom"ioredis";constmetadataCache=newRedisCache(newRedis(process.env.REDIS_URL));constgroup=awaitopen(store,"",{ metadataCache });
RedisCache also accepts a connection URL directly
(new RedisCache("redis://..."), with optional ioredis options as a second
argument); the client is then created lazily on first use. Passing a
pre-configured client is preferred — with a bare URL, ioredis defaults apply
and commands issued while Redis is unreachable can stall before the store
fallback kicks in.
Cache keys are scoped as ${storeId}:${metadataKey}. The store identity is
derived automatically for S3Store and HTTPStore; for any other store you
must pass an explicit storeId, otherwise open throws immediately
(preventing silent per-pod key divergence):
awaitopen(customStore,"",{ metadataCache,storeId: "my-dataset-v1"});
Observability hooks
Every layer accepts an optional per-instance observability object — no
global registry. The same object can be passed to multiple layers; each layer
fires only the events it owns:
import{S3Store,CachedStore,open}from"@i4sea/zarr-node";constobservability={onCacheHit: ({ tier, key })=>metrics.inc(`cache.hit.${tier}`),// "memory" | "disk" | "shared"onCacheMiss: ({ tier, key })=>metrics.inc(`cache.miss.${tier}`),onStoreFetch: ({ key, bytes, latencyMs })=>metrics.observe("store.fetch_ms",latencyMs),onRetry: ({ attempt, status, error })=>logger.warn(`retry ${attempt} status=${status}`),onChunkDecoded: ({ bytes, codec, decodeMs })=>metrics.observe(`decode.${codec}`,decodeMs),onInFlightBytes: (current)=>metrics.gauge("inflight_bytes",current),onMissingChunk: ({ key })=>logger.error(`missing chunk ${key}`),};// Store layer: onStoreFetch, onRetryconststore=newS3Store({ bucket, region, observability });// Disk-cache layer: onCacheHit/onCacheMiss (tier "disk")constcached=newCachedStore(store,{ cacheDir, maxSizeBytes, observability });// Open path: onCacheHit/onCacheMiss (tier "shared", with metadataCache)constgroup=awaitopen(cached,"",{ metadataCache, observability });// Read path: memory-tier hit/miss, onChunkDecoded, onInFlightBytes, onMissingChunkconstdata=awaitarray.get(selection,{ observability });
A throwing (or rejecting) handler is swallowed and never breaks a read. When
no hooks are registered there is zero overhead — payload objects are not even
allocated.
Offloading decompression (worker threads)
Blosc decode is synchronous CPU work (it runs on WASM), so a large chunk blocks
the event loop for the whole decode — degrading the latency of every other
request in a shared API pod. gzip/zlib already run on the libuv threadpool
and are unaffected.
Opt in by passing a DecodePool via decodeWorkers. Chunks whose compressor is
offloadable (currently Blosc) and whose compressed size is at least minBytes
are decoded on a worker thread; everything else decodes inline as before. Create
one pool per process, reuse it across reads, and call terminate() on shutdown
(idle workers keep the process alive).
import{DecodePool,open}from"@i4sea/zarr-node";constdecodeWorkers=newDecodePool({poolSize: 4,// default: availableParallelism() - 1minBytes: 256*1024,// skip offload below this compressed size (IPC isn't worth it)});constarray=awaitopen(store,"wind_vel");constdata=awaitarray.get(selection,{ decodeWorkers });// ... on shutdown:awaitdecodeWorkers.terminate();
The threshold is on the compressed size (known before decode). Use
onChunkDecoded (above) to measure decodeMs with and without the pool and
calibrate minBytes for your datasets; examples/benchmark-decode-workers.ts
runs that A/B and also reports event-loop lag.
Reference filesystem (kerchunk)
import{ReferenceStore,open}from"@i4sea/zarr-node";import{readFile}from"node:fs/promises";constmanifest=JSON.parse(awaitreadFile("output.json","utf-8"));conststore=newReferenceStore({spec: manifest});constarray=awaitopen(store,"temperature");constdata=awaitarray.read();
Spatial lookups (GridIndex)
@i4sea/zarr-node/spatial resolves a (lat, lon) to the nearest grid cell (i, j)
on a 2D curvilinear grid (e.g. a WRF domain). The grid is static per domain, so it
is loaded once and queried many times — each query is pure CPU.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constgrid=awaitGridIndex.fromGroup(group);// loads lat/lon onceconst{ i, j, distanceKm }=grid.nearest(-25.5,-44.5);constseries=await(awaitgroup.getArray("wind_vel")).get([null,[i,i+1],[j,j+1]]);
For ephemeral pods, persist the grid in a shared Cache (Redis) so only the first
pod pays the coordinate fetch — restarts and new pods rehydrate from the cache:
import{RedisCache}from"@i4sea/zarr-node/redis";constcache=newRedisCache(process.env.REDIS_URL!);// L1 (process) → L2 (Redis) → L3 (store). The key is derived per *domain*// (source_model/experiment/grid_id + shape), so every run of the same grid shares it.constgrid=awaitGridIndex.loadCached(group,{ cache });
Pass an explicit gridKey to control the cache key, or verifyGrid: true to fold a
corner sample of the coordinates into it (+2 cheap reads) when the dataset attrs
can't be trusted.
Polygon reads (readPolygon)
readPolygon streams — one time step at a time — only the cells geometrically
inside a lat/lon polygon of a [time, ...spatial] array. It reads each step as a
bounding-box block, so each backing chunk is fetched/decompressed at most once
(chunks typically span the full time axis and are reused across steps via a shared
MemoryCache), and peak memory stays bounded to ~one time slice regardless of the
time extent. Aggregation is the caller's concern — you get the raw in-polygon values.
import{openGroup}from"@i4sea/zarr-node";import{GridIndex,readPolygon}from"@i4sea/zarr-node/spatial";constgroup=awaitopenGroup(store);constarr=awaitgroup.getArray("t2m");// [time, ny, nx]constgrid=awaitGridIndex.fromGroup(group);// curvilinear lat/lonconstpolygon: Array<[number,number]>=[[-23.0,-43.5],[-23.0,-43.0],[-22.5,-43.0],[-22.5,-43.5],];forawait(conststepofreadPolygon(arr,{
polygon,spatialLayout: {kind: "2d", grid },})){// step.values: Float64Array of only the in-polygon cells for step.tconsole.log(step.t,step.values.length);}
resolvePolygonCells(arr, opts) returns the time-invariant selection
(cells + bbox + stride) without reading values — cells[k] aligns with
step.values[k]. Three coordinate layouts are supported:
{ kind: "1d", lat, lon } (monotonic axes), { kind: "2d", grid }
(curvilinear GridIndex), and { kind: "npoints", lat, lon } (unstructured
points). Set maxCells to cap huge selections with a clamped uniform stride
(reported as selection.stride; no default cap). A runnable example lives in
examples/read-polygon.ts.
Requirements
  • Node.js >= 22
  • ESM only ("type": "module")
API
Top-level functions
FunctionDescription
open(store, path?, options?)Open a Zarr array or group
openArray(store, path?, options?)Open a Zarr array (throws if not an array)
openGroup(store, path?, options?)Open a Zarr group (throws if not a group)
All three accept OpenOptions { metadataCache?, storeId?, metadataCacheTtlMs?, observability? }.
metadataCacheTtlMs sets a TTL (ms) on metadata-cache writes — use it with a
content-versioned storeId so obsolete versions' keys expire from a shared cache
instead of accumulating forever (omit ⇒ no expiry).
Store backends
ClassDescription
FileSystemStoreLocal filesystem
HTTPStoreHTTP/HTTPS with retry and timeout
S3StoreAWS S3 (requires @aws-sdk/client-s3)
CachedStoreWraps any store with disk caching
ReferenceStoreKerchunk JSON manifest
Caching
ClassDescription
CachedStoreDisk cache with LRU eviction and thundering herd protection
MemoryCacheIn-memory LRU cache for decoded chunks
InMemoryCacheIn-process Cache adapter for the metadata cache
RedisCacheRedis-backed Cache adapter (@i4sea/zarr-node/redis, requires ioredis)
Data classes
ClassDescription
ZarrArrayRead chunked array data with slicing support
ZarrGroupTraverse groups, list arrays, multi-array reads
Spatial
ClassDescription
GridIndexNearest (lat, lon) → (i, j) on a 2D grid, with optional Redis-backed grid cache (@i4sea/zarr-node/spatial)
readPolygon / resolvePolygonCellsStream / resolve the cells inside a lat/lon polygon of a [time, ...spatial] array (@i4sea/zarr-node/spatial)
Contributing
See CONTRIBUTING.md.
License
MIT
About
Read-only Zarr v2 array reader for Node.js
Resources
Readme
MIT license
Code of conduct
Code of conduct
Contributing
Contributing
Security policy
Security policy
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages