[v3] Structured dtype support

[jhamman]

Zarr-Python 2 supported structured dtypes. I don't think this has been discussed much, if at all, in the context of version 3.

This once worked:

a = np.array(
 [(b"aaa", 1, 4.2), (b"bbb", 2, 8.4), (b"ccc", 3, 12.6)],
 dtype=[("foo", "S3"), ("bar", "i4"), ("baz", "f8")],
)
za = zarr.array(a, chunks=2, fill_value=None)
za[:]

[d-v-b]

The v3 spec lists a finite set of dtypes and structured dtypes are not explicitly in the list. To be spec compliant and support structured dtypes, I see two options:

• express a structured dtype via the raw bits r* dtype. This seems unsatisfactory, because two structured dtypes might have the same raw bits representation but very different semantics (e.g., a pair of float64s vs a pair of int64s will look the same as raw bits)
• define a spec extension for structured dtypes. I don't think this would be a technical challenge, but it would be important to get consensus across implementations about the JSON serialization for the dtype. We would not want to overfit to numpy conventions. This seems like a better option, but it requires some social logistics.

[ilan-gold]

@d-v-b @jhamman Has there been any movement on this? I see there is a comment in the code base about this:

zarr-python/tests/test_indexing.py

Line 161 in 2fe12a7

# todo: uncomment the structured array tests when we can make them pass,

This would be something of a regression for anndata as a project if we couldn't support this (or we'd have to go with option 1 and investigate the current status with readers in other languages to see if they could be made to support this functionality).

[jhamman]

@ilan-gold - no progress and I agree its a regression. I think we should support it for v2 arrays but until there is an extension dtype for this in v3, we'll likely need to leave it out.

[ilan-gold]

That would be very fine with me. We'll need to "break" things for the v3 format anyway, so I don't have strong opinions there.

[d-v-b]

but until there is an extension dtype for this in v3

I do think we can assume that the v3 spec process will ultimately provide a clear framework for adding new dtypes, so it might be useful to think about basic questions like how these types should be named / parametrized in JSON.

[d-v-b]

we would probably start with the v2 conventions, and then consider how to ensure that a v3 version of that isn't too numpy-specific.