Add support for structured dtypes to `zarr3` driver, open structs as void by BrianMichell · Pull Request #271 · google/tensorstore

BrianMichell · 2026-01-05T15:22:04Z

Supersedes #264

Resolves comments 1 and 2

…river

…t for raw bits dtype

Implement shim for `open_as_void` driver level flag

* Begin removing void field shim * Fully removed void string shim * Cleanup debug prints * Remove shimmed validation * Remove unnecessary comment * Prefer false over zero for ternary clarity

* Implement a more general and portable example set * Fix driver cache bug * Update example for template * Cleanup example * Remove testing examples from source

* Use the appropriate fill value for open_as_void structured data * Cleanup

laramiel · 2026-01-22T23:18:00Z

I'll try to get to this in about a week, before I look this one over, please double check that the prior PR works for you. Also look over this one and see if any of the suggestions from the other one applies.

Matches the pattern from zarr v2 driver (PR google#272). When both "field" and "open_as_void" are specified in the spec, return an error since these options are mutually exclusive - field selects a specific field from a structured array, while open_as_void provides raw byte access to the entire structure.

The zarr3 URL syntax cannot represent field selection or void access mode. Following the pattern from zarr v2 driver (PR google#272), ToUrl() now returns an error when either of these options is specified instead of silently ignoring them.

…trip Following the pattern from zarr v2 driver (PR google#272), override GetBoundSpecData in ZarrDataCache to set spec.open_as_void from ChunkCacheImpl::open_as_void_. This ensures that when you open a store with open_as_void=true and then call spec(), the resulting spec correctly has open_as_void=true set. Without this fix, opening a store with open_as_void=true and then getting its spec would lose the open_as_void flag, causing incorrect behavior if the spec is used to re-open the store.

BrianMichell · 2026-05-21T14:48:13Z

Looks like development has picked back up on Tensorstore. I think I've addressed any outstanding issues and made sure everything is following the defined spec. Please let me know if there's anything else I can do to help move this forward any more!

laramiel · 2026-05-21T20:59:39Z

@jbms and I talked briefly about this and he has some comments; briefly there are a few concerns regarding endian encoding/decoding when used with the zarr3 dtype and with field selection.

edit: ... making the "bytes" codec understand the new ZarrDType might be a somewhat invasive change to how the codecs work.

Use iota instead of looping

google#271 (comment)

BrianMichell added 24 commits November 24, 2025 20:55

Begin examining how to best add structured array support to Zarr v3 d…
44b4473
…river

Merge branch 'google:master' into v3_structs
e6df164

Updates to have proper reads
187f424

Local testing and examples
c2e73cd

Begin adding support for opening struct arrays as void and add suppor…
9e8ed94
…t for raw bits dtype

Fix failing tests
44c765e

Resolve issues with opening struct as void
547642d

Remove debug print
2a4c3d8

Add field for open as void
b0abb94

Add a shim for new open_as_void flag open option
fff0a5b

Revert some formatting changes
b6c24f9

revert gitignore changes
488b160

Merge pull request#1from BrianMichell/v3_structs_compatibility
537d309
Implement shim for `open_as_void` driver level flag

V3 structs remove shim (#2)
54941a0
* Begin removing void field shim * Fully removed void string shim * Cleanup debug prints * Remove shimmed validation * Remove unnecessary comment * Prefer false over zero for ternary clarity

Fix structured fill value population
c9f58f9

V3 examples merge (#3)
7655cfd
* Implement a more general and portable example set * Fix driver cache bug * Update example for template * Cleanup example * Remove testing examples from source

Remove vestigial example build
8c4c4ca

V3 structs fix fills (#4)
4b590f8
* Use the appropriate fill value for open_as_void structured data * Cleanup

Merge branch 'google:master' into v3_structs
7691c83

Add new options to schema
c0082a0

Fix copyright header date
9a46c82

Cleanup (#5)
b9b5e41

Merge zarr3 structured dtype support
d5f6201

Remove default values
31e55ec

BrianMichell mentioned this pull request Jan 5, 2026
Add support for structured dtypes to zarr3 driver, open zarr 2 and 3 structs as void #264
Closed

BrianMichell added 4 commits January 23, 2026 15:05

Merge branch 'google:master' into v3_structs_and_void
aad0ee0

Merge branch 'master' into v3_structs_and_void
64681c5

laramiel reviewed May 21, 2026
View reviewed changes

Comment threadtensorstore/driver/zarr3/codec/transpose.cc Outdated

laramiel reviewed May 21, 2026
View reviewed changes

Comment threadtensorstore/driver/zarr3/codec/codec_spec.h

laramiel reviewed May 21, 2026
View reviewed changes

Comment threadtensorstore/driver/zarr3/dtype.cc

laramiel reviewed May 21, 2026
View reviewed changes

Comment threadtensorstore/driver/zarr3/dtype.cc

laramiel reviewed May 21, 2026
View reviewed changes

Comment threadtensorstore/driver/zarr3/dtype.h

BrianMichell added 7 commits May 22, 2026 13:36

Resolvegoogle#271 (comment)
84fc385
Use iota instead of looping

Resolvegoogle#271 (comment)
0a4238a

Merge branch 'master' into v3_structs_and_void
e19f608

Resolvesgoogle#271 (comment)
ce5199e
google#271 (comment)

Resolvegoogle#271 (comment)
d5de2bd

Merge branch 'master' into v3_structs_and_void
f991259

Address codec encoding/decoding concerns and fix endian validation bug
c713fa2