Zarrquet implementation: Adding Zarrs-based TableProvider with bindings in Python. by alxmrs · Pull Request #69 · xqlsystems/xarray-sql

alxmrs · 2025-07-17T14:43:12Z

Initial implementation of #4. Rust sources were mostly generated by Claude Code.

…le with `uv build`.

…batches

…e duplicate logic.

…some refactoring.

…ll investigate later.

…r, with unit tests.

…est base.

…iew.

…illegible.

…to corrent column names as well as correct scanning (type handing).

… to make sure they're properly testing the bindings.

alxmrs

First part of a self-code review (for the benefit of Claude). Will resume tomorrow.

alxmrs · 2025-07-29T19:17:09Z

+ DataType::Int64 => {
+ Arc::new(Int64Array::new_null(0)) as Arc<dyn arrow_array::Array>
+ }
+ DataType::Float64 => {
+ Arc::new(Float64Array::new_null(0)) as Arc<dyn arrow_array::Array>
+ }
+ DataType::Int32 => {
+ Arc::new(Int32Array::new_null(0)) as Arc<dyn arrow_array::Array>
+ }
+ DataType::Float32 => {
+ Arc::new(Float32Array::new_null(0)) as Arc<dyn arrow_array::Array>
+ }
+ DataType::Utf8 => Arc::new(arrow_array::StringArray::new_null(0))
+ as Arc<dyn arrow_array::Array>,
+ _ => Arc::new(Int64Array::new_null(0)) as Arc<dyn arrow_array::Array>, // Default fallback


I think there are missing data types here.

…atch rust!!

alxmrs · 2026-02-01T21:03:52Z

This is better implemented in arrow-zarr or simply #100.

alxmrs added 22 commits July 14, 2025 22:06

Integrated rust into the project via maturin
6f9fd56

Properly set up the rust project (with dummy table provide, can compi…
fd6fe0e
…le with `uv build`.

Phase 1 and 2 implementation of zarr store plan!
4dcaa52

Next step: only processes a single array in the zarr group as record …
3aef11a
…batches

Attempting to minimize copying.
ff75797

Had Claude refactor the code to introduce generics, in order to reduc…
3881787
…e duplicate logic.

Better explained requirements for tables, claude wrote tests.
cbfc0e6

Applied automatic Zarr fixes from the compiler.
97e7a3c

Core data var to table logic implemented, according to Claude. Needs …
baf4f3b
…some refactoring.

Adding test_data to gitignore.
09251fb

Added example and test scripts as sanity checks. might delete later.
df8debd

Adding predicate pushdown logic, but I think it may need revision, wi…
b55db35
…ll investigate later.

Generated integration tests to test sql functionality.
b6efef2

Initial from_zarr implementation, which calls the rust TableProvide…
9e6a0b0
…r, with unit tests.

First stage of multi-stage test refactoring from Claude: creating a t…
1937393
…est base.

Made some minor refactors.
6457dd8

Implemented strategy 2 and 3.
0777064

Import opt.
7a5700a

Tests for semantic equivalence to xarray operations.
624c3af

Moved test data creation scripts into examples.
7c328e0

Removing examples for now, because they add too much noise to the rev…
028e69b
…iew.

Reformatted the python code with pyink.
b30bde6

alxmrs commented Jul 17, 2025
View reviewed changes

Comment threadsrc/table_provider.rs Outdated

alxmrs added 5 commits July 17, 2025 08:08

Import optimization
fff4203

Revert to simpler tests! These tests have gotten too complicated and …
0dd67fd
…illegible.

WIP, manually refactoring tests to test both from_zarr and from_dataset
cd59d51

I think I found an actual error with the Zarr logic.
27baf67

Fixed signficant correctness bugs in the tabel provider impl related …
38f2796
…to corrent column names as well as correct scanning (type handing).

alxmrs changed the title ~~Zarrquet implementation: Adding Zarrs based TableProvider with bindings in Python.~~Zarrquet implementation: Adding Zarrs-based TableProvider with bindings in Python.Jul 17, 2025

Test is incorrect; updating.
fdf19bf

alxmrs added 16 commits July 25, 2025 11:57

Called rust fmt (mimicking CI).
7b14abc

Passing rust lint checks and tests.
2fe9be4

Reformatted python with pyink.
e289bec

Renamed CI for clarity. Also, added a dev build step for python tests…
4bded87
… to make sure they're properly testing the bindings.

Adding maturin as a dev dep
e706eb6

Remove the s390x (IBM Mainframe) compiler target.
ffc6ebf

from_zarr is now only supported for Python 3.11+
7a3ca51

pyink fmt
43e5046

Try to fix linker problem for the maturin arm build.
92203fe

Put in the wrong file
7e20540

Upgrade ubuntu version to get gcc 14
29d132e

Attempting to fix IBM target via:rust-lang/stacker#79 (comment)
986b498

Add test to create test datasets for rust tests.
ea6870a

Adding integration tests from Claude.
b7a8494

Adding build step to run integration tests
69f065a

fmt int tests
243015b

alxmrs commented Jul 30, 2025
View reviewed changes

alxmrs mentioned this pull request Jul 31, 2025
What are the ultimate aims of arrow-zarr? datafusion-contrib/arrow-zarr#36
Closed

alxmrs added 2 commits September 11, 2025 13:57

Attempting to take a much simpler approach via an adapter.
14b4ea2

New, simpler approach that makes use of arrow-zarr.
5ac86e4

This was referenced Sep 26, 2025
Read subsets of Arrays in a Group into a single ZarrTable. datafusion-contrib/arrow-zarr#58
Closed
Added the ability to open a ZarrTable from a store path. datafusion-contrib/arrow-zarr#56
Closed

alxmrs added 4 commits September 26, 2025 23:48

Current WIP, got back to where I started :/
82e54cc

Fix: don't pass chunks.
7f24256

Made a really important fix! The python datafusion version needs to m…
1d58461
…atch rust!!

Updated xarray-sql branch of arrow-zarr.
76102d9

alxmrs mentioned this pull request Sep 27, 2025
Zarr to Datafusion ListingTable Adapter #20
Open

alxmrs mentioned this pull request Dec 19, 2025
Zarrquet: Create a referenced based filesystem to read virtual parquet from Zarr. #4
Open

alxmrs closed this Feb 1, 2026

Navigation Menu

Uh oh!

Zarrquet implementation: Adding Zarrs-based TableProvider with bindings in Python. - #69

Zarrquet implementation: Adding Zarrs-based TableProvider with bindings in Python.#69
alxmrs wants to merge 65 commits into
mainfrom
zarr-rs

alxmrs commented Jul 17, 2025•
edited
Loading

Uh oh!

Uh oh!

alxmrs left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alxmrsJul 29, 2025

Uh oh!

Uh oh!

Uh oh!

alxmrs commented Feb 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alxmrs commented Jul 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

alxmrs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alxmrsJul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alxmrs commented Feb 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alxmrs commented Jul 17, 2025•
edited
Loading