J language addon for Apache Arrow

Read (and eventually write) Apache Arrow and Parquet files to and from J. Uses C API.

Installation and Loading

1. Ensure that you have installed the Arrow GLib (C) Packages for your OS. Instructions can be found at: arrow.apache.org/install.

2. From your J session:

 install 'github:interregna/JArrow@main'
load 'data/arrow'

Usage

 install 'github:interregna/JArrow@main'
load 'data/arrow'
readParquetTable '~addons/data/arrow/test/test1.parquet'
┌─┬───────────────┐
│a│01234567│
├─┼───────────────┤
│b│87654321│
└─┴───────────────┘
readsParquetTable '~addons/data/arrow/test/test2.parquet'
┌────────┬──────────┬────────┬─────────┬───────┬────────┬───────┬───────┬────────┬────────┬────────┬──────────┬──────────┬───────────┬────────────┬─────────┬─────────┬───────┬───────────────┐
│Column 1│Column Two│shortCol│ushortCol│intcCol│uintcCol│int_Col│uintCol│int16Col│int32Col│int64Col│float32Col│float64Col│longlongCol│ulonglongCol│DoubleCol│StringCol│boolCol│datetime64Col │
├────────┼──────────┼────────┼─────────┼───────┼────────┼───────┼───────┼────────┼────────┼────────┼──────────┼──────────┼───────────┼────────────┼─────────┼─────────┼───────┼───────────────┤
│0 │ 100 │0 │0 │0 │100 │100 │100 │300 │500 │100 │ 600 │ 700 │100 │100 │ 100 │This │1 │946684800000000│
│1 │88.75 │1 │1 │1 │ 88 │ 90 │ 88 │263 │443 │ 88 │531.25 │613.75 │ 88 │ 88 │88.75 │ is │0 │946771200000000│
│2 │ 77.5 │2 │2 │2 │ 77 │ 80 │ 77 │227 │387 │ 77 │ 462.5 │ 527.5 │ 77 │ 77 │ 77.5 │all │0 │946857600000000│
│3 │66.25 │3 │3 │3 │ 66 │ 70 │ 66 │191 │331 │ 66 │393.75 │441.25 │ 66 │ 66 │66.25 │ valid │0 │946944000000000│
│4 │ 55 │4 │4 │4 │ 55 │ 60 │ 55 │155 │275 │ 55 │ 325 │ 355 │ 55 │ 55 │ 55 │text │1 │947030400000000│
│5 │43.75 │5 │5 │5 │ 43 │ 50 │ 43 │118 │218 │ 43 │256.25 │268.75 │ 43 │ 43 │43.75 │ │0 │947116800000000│
│6 │ 32.5 │6 │6 │6 │ 32 │ 40 │ 32 │ 82 │162 │ 32 │ 187.5 │ 182.5 │ 32 │ 32 │ 32.5 │data. │0 │947203200000000│
│7 │21.25 │7 │7 │7 │ 21 │ 30 │ 21 │ 46 │106 │ 21 │118.75 │ 96.25 │ 21 │ 21 │21.25 │ │0 │947289600000000│
└────────┴──────────┴────────┴─────────┴───────┴────────┴───────┴───────┴────────┴────────┴────────┴──────────┴──────────┴───────────┴────────────┴─────────┴─────────┴───────┴───────────────┘
readCSVTable '~addons/data/arrow/test/test1.csv'
┌──┬───────────────────────────...
│ID│1234581011121415 ...
├──┼───────────────────────────...
│y │100.669100.669100.669 100...
└──┴───────────────────────────...
NB. Note this is json-line format, not json-format. See: https://jsonlines.org
readsJsonTable'~Jaddons/data/arrow/test/test1.json'
┌───────┬──────────┐
│name │date │
├───────┼──────────┤
│Gilbert│12-13-2014│
│Alexa │09-04-1983│
│May │01-01-1924│
│Deloise│04-25-1894│
└───────┴──────────┘
readsFeatherTable '~addons/data/arrow/test/test1.feather'
┌────┬───┬──────┐
│team│pos│points│
├────┼───┼──────┤
│A │G │17 │
│A │F │17 │
│B │G │15 │
│B │F │ 5 │
│C │G │11 │
│C │F │10 │
│D │G │ 5 │
│D │F │14 │
└────┴───┴──────┘

(6!:16) and (6!:17) can be used to convert Arrow datetime64 types to and from ISO 8601 format (e.g. 2000-01-11T22:58:04). fromdate32 can be used to convert Arrow date32 types to YYYY M D tuples.

Notes

readsTable minimizes display time in the UI but uses more space

readTable minimizes space but can take more time to display

Development

1. In Jqt, identify your path for ~Projects jpath '~Projects'

2. Git clone the JArrow repo within ~Projects

3. Restart Jqt and open the Arrow project Project > Open > Projects > jarrow

4. Re-build the addon. Ctrl + F9

5. Run the addon. F9 (Re-build addon scripts, reload and run tests)

Examples: see test/test1.ijs

TODO

• Error catching for empty pointers, missing files, and general errors.
• Dereference / cleanup gobjects and allocated memory
• Additional data types
  • Dictionaries (need to store lookup tables)
  • Lists
  • Maps
• Tensors
• Documentation (see: ~/addons/gui/cobrowser/scriptdoc.ijs)
• CSV reader
• JSONL reader
• Arrow Feather (IPC v1) reader
• IPC files (".arrow" files) — readArrowTable, writeArrowTable
• IPC streams (".arrows" files) — readArrowsTable, writeArrowsTable
• Feather v2 writer — writeFeatherTable (alias of writeArrowTable)
• Parquet writer — writeParquet
• Flight client
• Flight server
• Non-local filesystems (S3)
• IPC streaming with event-driven calls

Writers (available with Arrow GLib 24.0.0)

 writeArrowTable tablePtr;'~out.arrow'NB. IPC file format
writeArrowsTable tablePtr;'~out.arrows'NB. IPC streaming format
writeFeatherTable tablePtr;'~out.feather'NB. Feather v2 (= IPC file)
writeParquet tablePtr;'~out.parquet'NB. Parquet

Unified surface (updated for Arrow GLib 24.0.0)

A small terse vocabulary giving JArrow a consistent handle-based columnar interface for J users.

NB. handle open / close / schema / read / project (gaps 1, 2, 7, 10)
h=: ho '~data.feather'NB. auto-detects format from extension
hs h NB. Arrow schema
tbl=: hr h NB. read full table
tbl=: 1000 hr h NB. read first 1000 rows
tbl=: ('price';'qty') hp h NB. project columns
hc h NB. closeNB. stream table fallback over .arrows files (gap 5)
sh=: sbo '~stream.arrows'
b=: sbn sh
sbc sh
NB. Arrow C ABI bridge — zero-copy hand-off (gap 3)'sa aa'=: ax tbl NB. pending per-type J→Arrow builders
tbl=: ai sa;aa NB. import the same pairNB. compute kernels via Arrow function registry (gap 6)
afsum col NB. sum kernel
afmean col / afmin col / afmax col / afcount col
'percentile_99' afk col NB. any registered kernel by nameNB. CSV / JSONL writers (gap 8)
tbl wcsv '~out.csv'
tbl wjsonl '~out.jsonl'NB. Custom extension types (gap 9)
extreg ''NB. register quant4 / quant8 / embed_f32
extq4 codes; scale; zero NB. decode quant4 to floats

The verb prefixes are the vocabulary:

prefix	category	gap
h	handle: open / read / close	1, 2, 7, 10
sb	stream batches	5
a	Arrow C ABI bridge	3
md	schema metadata	4
af	Arrow function (compute)	6
wcsv	CSV / JSONL write	8
ext	Custom extension types	9

Some verbs are skeletons that depend on extra GLib bindings. JArrow now attempts those optional binds at load time and leaves unsupported symbols inactive on older Arrow GLib installations.

sbo/sbn currently use a safe single-table fallback over readArrowsTable. The lower-level GLib read_next iterator crashes this J console build, so the public verb avoids that native path.

ax is intentionally guarded with a controlled assertion until the per-type J table to GArrowArray builders are implemented. This avoids the prior crash path through an unfinished read_next batch-reader bridge.

The IPC file (.arrow) and Feather v2 (.feather) formats are the random-access-preserving choice. .arrows (streaming, no footer) is the right shape for pipes and sockets. Parquet is the cross-ecosystem format.

Bug fixes in the Arrow GLib 24.0.0 refresh

• The IPC reader verbs readArrowTable, readArrowsTable, and readFileBufferTable were defined with =. (local) instead of =: (global) — the public-interface transfers block at the bottom of arrow.ijs silently failed to expose them. Now fixed; these verbs are reachable as documented.
• Added IPC writer wrappers; writer close is used as the flush point before deallocation.