feat: metadata-only support for storage transformers metadata

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b

[d-v-b]

Addresses #2178

This adds metadata-only support for the storage_transformers key in array metadata. By "metadata only" I mean that creating an array metadata document from JSON or a dict with a storage_transformers keyword will not error, and some very mild validation will be run (just ensuring that the value is an iterable or None), but the value is not used for any storage transforming, because we don't have any code for that yet.

But at least the metadata can be constructed and the storage_transformer value should round-trip through metadata properly. This should resolve#2178.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[zoj613]

If creating a metadata object from a JSON file that specifies a storage transformer list, then this implies reading from or writing to that array hinges on applying the transformer pipeline since it may modify the keys and/or values of that array node. I think ignoring it and issuing a warning is not enough since reading from said array using the metadata would potentially result in the wrong byte sequence being passed to its codec pipeline (as a result of skipping the transformation pipeline).

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

The case where a storage_transformer field could be safely ignored when parsing a JSON is when it is an empty list, as indicated by the spec

[d-v-b]

To me this appears as a programming error. I think parsing a metadata document with a specified storage_transformers field should raise an exception to indicate that zarr-python cannot reliably read such an array node since it was clearly created by an implementation that used a storage transformer pipeline.

I think this is valid, but we should raise when constructing an Array from metadata that references a storage transformer, rather than when constructing the array metadata itself. I will implement this.

[jhamman]

after thinking about this some more, I believe we should raise an error here instead of warning.

We allow:

• metadata w/o the storage_transformers key
• metadata with storage_transformers == [] or None

We error for:

• metadata w/ len(storage_transformers) > 0

[d-v-b]

why not error when a user tries to create an array with the metadata? I'm thinking about how someone would develop a storage transformer with zarr-python as a dependency. If the metadata parsing functionality supports storage transformers, then they can use that immediately, and then subclass / implement their own array class that can use the storage transformer. Seems like a better workflow than forcing them to also subclass the metadata class?

[d-v-b]

The changes in this PR now result in errors if an AsyncArray is created from ArrayV3Metadata that contains a non-zero number of storage transformers. However, you can create metadata with storage transformers without any error or warnings.

[jhamman]

Thanks @d-v-b