Implement Zarr V3 protocol by grlee77 · Pull Request #898 · zarr-developers/zarr-python

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 · 2021-12-01T04:05:52Z

This PR builds on #897, adding zarr v3 support ~~to most of the high-level convenience functions (I haven't looked at the consolidated metadata yet).~~

EDIT: I closed the prior v3 PRs leading up to this one as their commits are also included here along with additional bug fixes and tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

codecov · 2021-12-01T04:17:18Z

Codecov Report

Merging #898 (7aa5dff) into master (b98c799) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@ Coverage Diff @@## master #898 +/- ##
=========================================
Coverage 99.94% 99.95% =========================================
Files 32 33 +1 Lines 11256 13622 +2366 =========================================
+ Hits 11250 13616 +2366 
Misses 6 6

Impacted Files	Coverage Δ
zarr/_storage/absstore.py	`100.00% <100.00%> (ø)`
zarr/_storage/store.py	`100.00% <100.00%> (ø)`
zarr/attrs.py	`100.00% <100.00%> (ø)`
zarr/convenience.py	`100.00% <100.00%> (ø)`
zarr/core.py	`100.00% <100.00%> (ø)`
zarr/creation.py	`100.00% <100.00%> (ø)`
zarr/hierarchy.py	`100.00% <100.00%> (ø)`
zarr/meta.py	`100.00% <100.00%> (ø)`
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_attrs.py	`100.00% <100.00%> (ø)`
... and 11 more

pep8speaks · 2021-12-15T00:52:47Z

Hello @grlee77! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-11 03:47:51 UTC

zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

classmethods adapted from zarrita code

avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

update hexdigests

No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

grlee77 · 2022-03-04T19:05:46Z

NestedDirectoryStoreV3 has been removed as discussed above: #898 (comment)

joshmoore

Looking nice, @grlee77! A number of mostly aesthetic points below after an initial read. Don't think any of them are MUSTs before merging this into a dev branch. (See zarr-developers/governance#15)

The only other high-level comment, also aesthetic, is whether zarr._storage.v3 (or similar) would be appropriate earlier rather than later.

joshmoore · 2022-03-07T20:45:06Z

 self.rmdir()
+
+
+def rmdir_abs(store: ABSStore, path=None):


Why is this not a method on the store?

Not sure why I did it that way. I can make this ABSStore.rmdir and just call it from there in ABSStoreV3.rmdir

joshmoore · 2022-03-07T20:52:36Z

 del store[key]


+def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:


Knowing how this module is growing, is there any place relatively convenient that we could "hide" v3-specific methods?

joshmoore · 2022-03-07T20:56:33Z

+ key == 'zarr.json')
+
+ # cannot create a group without a path in v3
+ # so create /meta/root/consolidated group to store the metadata


Use of "/meta/root/consolidated" likely needs review at the zarr-specs level.

I opened zarr-developers/zarr-specs#136 so we don't forget

joshmoore · 2022-03-07T20:59:59Z

- store = _normalize_store_arg(store)
+ store = _normalize_store_arg(store, zarr_version=zarr_version)
+ if zarr_version is None:
+ zarr_version = getattr(store, '_store_version', 2)


Worth planning now for when the default swaps from 2 to 3? Either by using a constant or by documenting the locations while you have it in mind?

Okay, I added DEFAULT_ZARR_VERSION (currently set to 2) in place of 2 in these checks.
It is intended as a constant used by developers, not something a user would set in their code to select the default version.

These define the root path for metadata and data, respectively

Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

… to 3

…a helper

joshmoore · 2022-03-15T18:27:26Z

Anything else planned from your side, @grlee77 ?

grlee77 · 2022-03-15T19:08:58Z

Anything else planned from your side, @grlee77 ?

Before release we will need to decide on where we want the import path for v3 Stores to be (currently in the same file as the V2 ones). We can think about that and a possible environment variable to enable/disable v3 visibility in a followup, though.

joshmoore · 2022-03-23T23:21:08Z

👍 for handling the refactoring and v3-switch in follow ups. As discussed during the past two community calls, getting this in to the mainline. 🎉 (And as promised for @grlee77: 🍪)

Clear warning for @zarr-developers/core-devs : the mainline is now in an unreleasable v3 state. If 2.11.x needs to go out, a dedicated v2 branch will need to be forked from a point before this merge.

See zarr-developers/governance#15 for some background.

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

grlee77 added 5 commits November 17, 2021 13:30

add v3 store classes
64bbbd8
Define the StoreV3 class and create v3 versions of most existing stores Add a test_storage_v3.py with test classes inheriting from their v2 counterparts. Only a subset of methods involving differences in v3 behavior were overridden.

add TODO comment to meta.py
85b8e23

Merge remote-tracking branch 'upstream/master' into v3-store-part1
5fee6fb

fix flake8 errors
983d190

Merge remote-tracking branch 'upstream/master' into v3-store-part1
4057088

grlee77 added 3 commits December 14, 2021 19:15

follow zarr v3 spec when dealing with extension data types
9ed6181

fixes to v3 dtype handling
662e310

Merge remote-tracking branch 'upstream/master' into v3-store-part1
9950a4d

grlee77 force-pushed the v3-store-part6 branch from 49163e0 to 0540124Compare December 15, 2021 00:52

flake8 cleanup
450c575

grlee77 force-pushed the v3-store-part6 branch from 0540124 to b13a6b3Compare December 15, 2021 14:48

remove duplicate lines in Metadata2.encode_array_metadata
63a9e3c

grlee77 force-pushed the v3-store-part6 branch 2 times, most recently from de243b2 to 266ab58Compare December 16, 2021 20:43

Fix fields in array metadata
1a2a1ef
zarr_version should not be in the array metadata, only the base store metadata compressor should be absent when there is no compression

grlee77 force-pushed the v3-store-part6 branch from 266ab58 to 99d9632Compare December 17, 2021 04:00

grlee77 added 2 commits December 16, 2021 23:36

Fix encode/decode of codec metadata
bcb2d5f
classmethods adapted from zarrita code

add missing level to Zlib in _decode_codec_metadata
9bfeacb

grlee77 force-pushed the v3-store-part6 branch from 99d9632 to 37858e1Compare December 17, 2021 05:37

add extensions entry to v3 array metadata
23aca42

grlee77 force-pushed the v3-store-part6 branch from 37858e1 to 49a42b5Compare December 17, 2021 05:57

grlee77 added 6 commits December 17, 2021 01:17

dimension_separator should not be in the array metadata for v3
e47035d

update Attributes, adding StoreV3 support
7549d5b
avoid pytest error about missing fixture fix flake8 error related to zarr_version fixture

add StoreV3 support to core Array object
765fe7d

update hexdigests
5efc786

handle additional codecs that were not implemented in zarrita
6a308c8
update hexdigests

fix
f7883fc

grlee77 force-pushed the v3-store-part6 branch from 49a42b5 to 60e3560Compare December 17, 2021 06:19

remove NestedDirectoryStoreV3
9f56e82
No need for this class as DirectoryStoreV3 with / chunk separator can be used instead

flake8
88405a8

sanketverma1704 mentioned this pull request Mar 5, 2022
Changing the default branch of Zarr-Python zarr-developers/governance#15
Closed

joshmoore reviewed Mar 7, 2022
View reviewed changes

grlee77 added 6 commits March 8, 2022 20:54

Merge remote-tracking branch 'upstream/master' into v3-store-part6
87b9543

remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3
69e5a12

define meta_root and data_root variables
58c8c20
These define the root path for metadata and data, respectively

move _valid_key_characters to be a StoreV3 class field
5e1ed7f

make _get_hierarchy_metadata strictly require 'zarr.json'
3593cad
Still use a default set of metadata in __init__ method of Group or Array classes. Add a _get_metadata_suffix helper that defaults to '.json' if metadata is not present.

ignore type checks for _get_metadata_suffix
f036215

grlee77 mentioned this pull request Mar 11, 2022
consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
Open

grlee77 added 4 commits March 10, 2022 21:59

Merge remote-tracking branch 'upstream/master' into v3-store-part6
6f08185

remove unneeded if/else in Array and Hierarchy class __init__
353058d
default metadata already gets added by Metadata3.encode_hierarchy_metadata when meta=None

remove unused import
2ec2b2d

define DEFAULT_ZARR_VERSION so we can later more easily change from 2…
2b82967
… to 3

grlee77 mentioned this pull request Mar 11, 2022
recommended file extension for zarr-v3 stores (.zr3?) zarr-developers/zarr-specs#137
Closed

add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadat…
7aa5dff
…a helper

joshmoore merged commit 2c13b95 into zarr-developers:masterMar 23, 2022

joshmoore mentioned this pull request Apr 7, 2022
Create fsstore from filesystem #911
Merged
6 tasks

jstriebel mentioned this pull request Dec 9, 2022
Sync v3 implementation with spec #1290
Closed

jhamman mentioned this pull request Sep 6, 2023
Remove chunk_memory_layout from V3 metadata in fafor of transpose codec #1517
Closed

		del store[key]


		def _rmdir_from_keys_v3(store: StoreV3, path: str = "") -> None:

Uh oh!

Uh oh!

Conversation

grlee77 commented Dec 1, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecovBot commented Dec 1, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pep8speaks commented Dec 15, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2022-03-11 03:47:51 UTC

Uh oh!

grlee77 commented Mar 4, 2022

Uh oh!

joshmoore left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshmoore commented Mar 15, 2022

Uh oh!

grlee77 commented Mar 15, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshmoore commented Mar 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

grlee77 commented Dec 1, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecovBot commented Dec 1, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pep8speaks commented Dec 15, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2022-03-11 03:47:51 UTC

Uh oh!

grlee77 commented Mar 4, 2022

Uh oh!

joshmoore left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

grlee77 commented Dec 1, 2021•
edited
Loading

codecovBot commented Dec 1, 2021•
edited
Loading

pep8speaks commented Dec 15, 2021•
edited
Loading

grlee77 commented Mar 15, 2022•
edited
Loading

grlee77 commented Dec 1, 2021•
edited
Loading

codecovBot commented Dec 1, 2021•
edited
Loading

pep8speaks commented Dec 15, 2021•
edited
Loading

grlee77 commented Mar 15, 2022•
edited
Loading

grlee77 commented Dec 1, 2021•
edited
Loading

codecovBot commented Dec 1, 2021•
edited
Loading

pep8speaks commented Dec 15, 2021•
edited
Loading