Open zip store from path with zip suffix

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?

[aulemahal]

• Opening a string or Path path is redirected to ZipStore if the path has a .zip suffix.

Fixes#2831

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

The added test is not as elegant as the other from the suite. Is there a way to get an existing zipped zarr dataset within the test suite ? Should I create another fixture instead of writing my own zarr in the test ?

[aulemahal]

I did not add anything in the user-guide/*.rst files as I didn't feel it was necessary. The docstring of open( already makes mention of this feature.

[dstansby]

Looks good! Just one minor comment, but I think worth fixing 👍

[aulemahal]

Hi @dstansby , I'm not sure, but it seems your suggested change that worked in march now breaks the tests. Did some assumptions change about mode=None ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (ed60e13) to head (8c234da).

Additional details and impacted files

@@ Coverage Diff @@## main #2856 +/- ##
=======================================
Coverage 93.47% 93.47% =======================================
Files 90 90 Lines 11967 11970 +3 =======================================
+ Hits 11186 11189 +3 
Misses 781 781 

Files with missing lines	Coverage Δ
src/zarr/storage/_common.py	89.67% <100.00%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aulemahal]

Hi @dstansby , I updated the PR with the current main.

To justify coming back to this PR after almost one year:

In between the moment I opened this PR and now, I saw there was the objective of ZEP8 which would have been another way of implementing this convenience. However, this project seems to have been stalled for the last 6 months.

My organisation has a lot of zipped zarrs in its database. We are running a lot of code on a traditionnal HPC which has a relatively low limit on inodes. Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes. Xarray not being able to read zipped zarr directly is an issue that blocks us from upgrading to Zarr 3.

We have a few solutions:

• Having zarr do it natively : the easiest one, this PR
• A custom xarray backend + changes in intake-esm : more effort, less elegant, needs development in a low-activity repo.
• Monkeypatch zarr : extra ugly, much less nice than contributing to open source software
• Flip to netCDF / unzipped zarr : lots of work to rewrite everything, great loss in performance on the HPC

I would understand if my patch here is not the most elegant way forward in the grand scheme of Zarr. But, I still think this could be useful for more than us. It could be seen as a "temporary solution" before ZEP8 is implemented fully.

[d-v-b]

Zipped zarr offer much better performance than netCDFs and a single node. Local zarrs are more performant, but yield insane amounts of inodes.

I will look at this PR but in the meantime have you considered sharding? that gives you most of the benefits of zipping your data

[aulemahal]

Thanks!

Yes, we did, but not that much yet, we were a bit slow to begin to look into new Zarr v3 features. For a typical gridded climate datasets with a single data variable, you still have at least 3 inodes per variable (folder, data and json), which can easily yield 20x more inodes considering you have a few auxiliary coords, even if writing the data in a single shard (not sure yet if this is good or not for multi-GB dataseets). So sharding could be a compromise indeed, but this is still a lot of inodes.

[ospinajulian]

Hi, is there any update on this PR?