icechunk-js

Read-only JavaScript/TypeScript reader for Icechunk repositories, designed for use with zarrita.

• Pure TypeScript, works in browsers and Node.js 18+
• Icechunk v1 and v2 format auto-detection
• All chunk payload types: inline, native, and virtual

Getting Started

npm install icechunk-js

Basic Usage with zarrita

import{IcechunkStore}from"icechunk-js";import{open,get}from"zarrita";// Open a store from a URLconststore=awaitIcechunkStore.open("https://bucket.s3.amazonaws.com/repo");// Open an array and read dataconstarray=awaitopen(store.resolve("/temperature"),{kind: "array"});constdata=awaitget(array,[0,0,null]);

For Development

npm install
npm run dev
npm run test
npm run typecheck

To regenerate FlatBuffers TypeScript after syncing schemas, run npm run generate:fbs. The pinned flatc compiler is downloaded automatically if not already available. The version is set in scripts/ensure-flatc.sh.

API

IcechunkStore

The main class for zarrita integration. Implements zarrita's AsyncReadable interface with both get() and getRange() (needed for sharded arrays). Pass zarrita's withRangeCoalescing to coalesce concurrent reads against the same backing object. This requires zarrita >= 0.7.

Note: Range coalescing uses zarrita's merged abort-signal behavior. If one read in a merged batch is aborted, other reads in the same batch may also reject. Avoid sharing an AbortController across requests that must cancel independently.

import{IcechunkStore}from"icechunk-js";import{withRangeCoalescing}from"zarrita";// Open from a URL (default: branch "main")conststore=awaitIcechunkStore.open("https://example.com/repo",{branch: "main",// tag: 'v1.0',// snapshot: 'ABC123...',// formatVersion: 'v1', // skip format auto-detection for v1 repos// maxManifestCacheSize: 50, // LRU cache size (default: 100)// withRangeCoalescing, // opt into merged range reads// signal: abortController.signal, // cancel initialization// validateChecksums: true, // integrity headers for virtual chunks// azureAccount: 'myaccount', // required for az:// virtual chunks});// Open from an existing ReadSessionconststore=awaitIcechunkStore.open(session);// Open from a custom Storage backendconststore=awaitIcechunkStore.open(myStorage,{branch: "main"});

Store methods

// Scope to a subpath (shares the same session and cache)constscoped=store.resolve("group/subgroup");// Browse the hierarchyconstchildren=store.listChildren("/");// direct children of rootconstallNodes=store.listNodes();// all nodes in the snapshotconstnode=store.getNode("/temperature");// single node by pathconstmeta=store.getMetadata("/temperature");// parsed zarr.json// Access the underlying session for advanced operationsconstsession=store.session;

Virtual chunk authentication

For private datasets with virtual chunks (S3, GCS, Azure), provide a fetchClient that handles authentication:

importtype{FetchClient}from"icechunk-js";constfetchClient: FetchClient={asyncfetch(url,init){// URL rewriting, pre-signing, or header injection happens here.// icechunk-js has already translated s3:// → https:// and built// Range headers in `init` (plus If-Match when validateChecksums is on).constsignedUrl=awaitpresign(url);returnglobalThis.fetch(signedUrl,{
...init,headers: { ...init?.headers,Authorization: `Bearer ${token}`},});},};conststore=awaitIcechunkStore.open("https://example.com/repo",{
fetchClient,});

Cloud storage URLs in virtual chunk references are automatically translated:

• s3://bucket/key → https://bucket.s3.amazonaws.com/key
• gs://bucket/key (or gcs://) → https://storage.googleapis.com/bucket/key
• az://container/path (or azure://) → https://{azureAccount}.blob.core.windows.net/container/path
• abfs://container@account.dfs.core.windows.net/path → https://account.blob.core.windows.net/container/path

For S3, addressing follows the repo's virtual-chunk-container config (region, endpoint, path-style), mirroring the Rust implementation. Buckets whose name contains a dot must use path-style addressing; when the container config records a region (repos written by Icechunk do), reads go straight to that regional endpoint — which serves CORS and works in both Node and the browser, with no extra configuration.

If no region is known, dotted-name buckets fall back to the global endpoint, whose region redirect (for buckets outside us-east-1) is resolved automatically in Node but not in browsers — the cross-origin redirect carries no CORS headers. Supply a fetchClient that routes to the regional endpoint or a CORS-enabled proxy in that case.

Repository

For direct access to branches, tags, and checkouts.

Note: Over plain HTTP, listBranches() and listTags() only work reliably with v2 repos, which embed refs in the top-level repo file. For v1 repos, direct checkoutBranch() / checkoutTag() can work when the target ref still lives at the legacy ref.json path, but versioned ref filenames still require listPrefix() discovery, which HttpStorage does not provide. Use a listing-capable storage backend for full v1 branch/tag support.

import{Repository,HttpStorage}from"icechunk-js";// Replace with the root URL of a real Icechunk repository.conststorage=newHttpStorage("https://example.com/repo");// Auto-detect format (default)constrepo=awaitRepository.open({ storage });// Or with format version hint (skips /repo request for v1 stores)// const repo = await Repository.open({ storage, formatVersion: 'v1' });// List branches and tags (v2 repos, or storage backends that support listing)constbranches=awaitrepo.listBranches();consttags=awaitrepo.listTags();// Checkout to get a ReadSessionconstsession=awaitrepo.checkoutBranch("main");// or: repo.checkoutTag('v1.0')// or: repo.checkoutSnapshot('ABCDEFGHIJKLMNOP')

Walking commit history

forawait(constentryofrepo.walkHistory(session)){console.log(entry.id,entry.message,entry.flushedAt,entry.metadata);}

ReadSession

Low-level access to nodes, chunks, and snapshot metadata.

// Snapshot infoconstsnapshotId=session.getSnapshotId();constparentId=session.getParentSnapshotId();// null for rootconstmessage=session.getMessage();consttimestamp=session.getFlushedAt();constmetadata=session.getSnapshotMetadata();// Navigate the hierarchyconstnodes=session.listNodes();constchildren=session.listChildren("/group");constnode=session.getNode("/array");// Get Zarr metadata and chunksconstzarrMeta=session.getMetadata("/array");constchunk=awaitsession.getChunk("/array",[0,0,0]);// Transaction log (what changed in this snapshot)consttxLog=awaitsession.loadTransactionLog();if(txLog){console.log("New arrays:",txLog.newArrays.length);console.log("Updated chunks:",txLog.updatedChunks.length);}

HttpStorage

HTTP/HTTPS storage backend using the Fetch API. Works in Node.js 18+ and browsers.

import{HttpStorage}from"icechunk-js";conststorage=newHttpStorage("https://bucket.s3.amazonaws.com/repo",{headers: {Authorization: "Bearer token"},credentials: "include",cache: "no-store",});

Custom Storage

Implement the Storage interface for other backends:

importtype{Storage,ByteRange,RequestOptions}from'icechunk-js';classMyStorageimplementsStorage{asyncgetObject(path: string,range?: ByteRange,options?: RequestOptions): Promise<Uint8Array>{ ... }asyncexists(path: string,options?: RequestOptions): Promise<boolean>{ ... }async*listPrefix(prefix: string): AsyncIterable<string>{ ... }}

License

MIT