top-level functions for reading, creating data by d-v-b · Pull Request #2463 · zarr-developers/zarr-python

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b · 2024-11-04T22:12:21Z

adds async and sync read_array, read_group, and read functions. read_array will read an array from storage. read_group does the same, but for groups, and read will try to read arrays or groups at the storage path. These functions wrap their respective creation functions, passing mode=r in each case.

I still need to add tests.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

d-v-b · 2024-11-04T22:15:26Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

d-v-b · 2024-11-05T13:07:25Z

in writing this PR I noticed that we don't have a create_array function, but we do have create_group. I don't like the asymmetry. We should fix that (by adding a create_array func).

I'm wrong, we don't have create_group! But we should! I think we should actually deprecate file-mode semantics across the board in favor of create_x, read_x functions.

…eat/read-funcs

d-v-b · 2024-11-05T16:47:38Z

This is ready for review. To summarize:

I created 5 new top-level exports:

read_array
read_group
read
create_array
create_group

The goal for these functions is to directly support common access patterns without forcing users to worry about the semantics of the mode keyword, and to guard against accessing zarr data with an unintended access mode (e.g., getting mutable access to an array you only want to read).

d-v-b · 2024-11-12T11:22:59Z

Some feedback from the community meeting last week:

One point of contention in this PR was the polymorphic read function, which returns either an array or group, depending on what's in the hierarchy. Personally I think this is invaluable for navigating unknown Zarr hierarchies, but others thought it was unnecessary or confusing. Does anyone else have thoughts?
Should this PR contain deprecation warnings for functions that are being made less essential by this PR (e.g., open and create)?

jhamman · 2025-01-02T22:22:02Z

Thanks @d-v-b and @normanrz - what a lift 🏋️‍♂️

…7489) # References and relevant issues Closes: #7484 # Description The latest --pre fail is due to recent changes in zarr released in 3.0.0rc1: zarr-developers/zarr-python#2463 JNI already has an issue and PR to fix upstream, so lets just block this rc.

d-v-b added 3 commits November 4, 2024 22:51

add functions for easy read-only data access
fd6ecd1

sync funcs
fa343f5

make read-only funcs top-level exports
d95eba8

d-v-b changed the title ~~Feat/read funcs~~functions for reading dataNov 4, 2024

Merge branch 'main' into feat/read-funcs
f6765bc

d-v-b mentioned this pull request Nov 5, 2024
Should we plan to deprecate File-mode-like semantics for accessing arrays and groups? #2466
Open

d-v-b added 3 commits November 5, 2024 17:38

add create_array, create_group, and tests
5d8445b

add top-level imports
9526571

Merge branch 'feat/read-funcs' of github.com:d-v-b/zarr-python into f…
90bf421
…eat/read-funcs

d-v-b marked this pull request as ready for review November 5, 2024 16:44

d-v-b added 2 commits November 5, 2024 17:52

add test for top-level exports
de280a7

add test for read
d9878cf

d-v-b changed the title ~~functions for reading data~~top-level functions for reading, creating dataNov 5, 2024

add asserts
e5217ce

d-v-b requested review from TomAugspurger and jhamman and removed request for jhamman November 5, 2024 17:33

d-v-b commented Nov 5, 2024
View reviewed changes

Comment threadsrc/zarr/core/array.py Outdated

d-v-b commented Nov 5, 2024
View reviewed changes

Comment threadsrc/zarr/core/group.py Outdated

d-v-b commented Nov 5, 2024
View reviewed changes

Comment threadtests/test_api.py Outdated

d-v-b added 3 commits November 5, 2024 18:38

Apply suggestions from code review
40cc7af

Merge branch 'main' into feat/read-funcs
d7ce58b

Merge branch 'main' into feat/read-funcs
a0dfe18

d-v-b requested a review from dstansby November 7, 2024 19:58

Merge branch 'main' into feat/read-funcs
98bc328

d-v-band others added 7 commits January 2, 2025 17:54

*params -> *like
4654cbd

*params -> *like, in tests
5cdb515

merge
0dc7dc6

Merge remote-tracking branch 'origin/main' into feat/read-funcs
c5c761e

adds deprecated compressor arg to Group.create_array
be60d73

Merge remote-tracking branch 'origin/main' into feat/read-funcs
ae1aa2a

docs
0a8b91c

jhamman approved these changes Jan 2, 2025
View reviewed changes

d-v-b mentioned this pull request Jan 2, 2025
default filters for v2 object dtype are wrong #2627
Closed

jhamman enabled auto-merge (squash) January 2, 2025 22:05

Merge branch 'main' into feat/read-funcs
315ba88

jhamman merged commit cc4dff9 into zarr-developers:mainJan 2, 2025

jhamman mentioned this pull request Jan 3, 2025
[v3] array creation: codecs #1943
Closed

jni mentioned this pull request Jan 3, 2025
Group.create_dataset does not match 2.x API #2631
Open

d-v-b mentioned this pull request Jan 3, 2025
add compressors, filters, serializer to Array as attributes #2633
Closed

jni mentioned this pull request Jan 4, 2025
Fix create_dataset with data kwarg #2638
Merged
6 tasks

psobolewskiPhD mentioned this pull request Jan 4, 2025
[Maint] Update version_denylist.txt to block zarr rc1 in --pre tests napari/napari#7489
Merged

This was referenced Jan 7, 2025
Fix zarr upstream tests pydata/xarray#9927
Merged
Group.array method ignores data argument #2667
Closed

jhamman mentioned this pull request Jan 8, 2025
Backwards compatibility for reading Array.filters and Array.codecs #2194
Closed

normanrz mentioned this pull request Jan 8, 2025
feat: change array creation signature to allow sharding specification [do not merge] #2169
Closed
6 tasks

jhamman mentioned this pull request Jan 10, 2025
Creating sharded arrays easily #2170
Closed

d-v-b deleted the feat/read-funcs branch February 5, 2025 13:06

d-v-b mentioned this pull request Feb 5, 2025
automatically specifying shards and chunks #2572
Closed

LDeakin mentioned this pull request Apr 8, 2025
(chore): update benchmarks zarrs/zarr_benchmarks#6
Merged

ilan-gold mentioned this pull request Apr 8, 2025
(bug): get_block_selection semantics #2971
Open

d-v-b mentioned this pull request Aug 11, 2025
explicitly control whether top-level array / group access routines can create #3364
Open

d-v-b mentioned this pull request Jul 10, 2026
Add two real datasets to the repo clbarnes/ome-zarr-rfc3-data#1
Closed