Feature/store list methods async generators by jhamman · Pull Request #1782 · zarr-developers/zarr-python

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

jhamman · 2024-04-10T04:21:40Z

Putting this PR up for discussion at tomorrow's refactor meeting.

on top of #1743

cc @d-v-b
TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

…ring issues

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

pep8speaks · 2024-04-10T04:21:50Z

Hello @jhamman! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/zarr/v3/group.py:

Line 313:83: W291 trailing whitespace
Line 319:101: E501 line too long (101 > 100 characters)
Line 320:101: E501 line too long (102 > 100 characters)

In the file src/zarr/v3/store/local.py:

Line 161:5: E303 too many blank lines (2)
Line 177:5: E303 too many blank lines (2)
Line 192:1: W293 blank line contains whitespace

In the file tests/v3/test_storage.py:

Line 59:29: E203 whitespace before ':'

Comment last updated at 2024-04-19 03:22:52 UTC

jhamman · 2024-04-10T04:22:06Z

+
+ async with aitercontext(children) as safe_children:
+ async for child in safe_children:
+ yield child


@d-v-b - take a look here.

…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

…p_children_fix

…ed to deal with

…and add some docstrings, and remove redundant def

…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

…_group_tests

… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

…ailing tests into subtests

…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

…ature/store-list-methods-async-generators

jhamman · 2024-04-19T04:21:19Z

closing in favor of d-v-b#110

d-v-band others added 23 commits March 26, 2024 21:04

feat: functional .children method for groups
e492be2

changes necessary for correctly generating list of children
b7f66c7

add stand-alone test for group.children
c7b333a

give type hints a glow-up
a64b342

test: use separate assert statements to avoid platform-dependent orde…
3d11fc0
…ring issues

test: put fixtures in conftest, add MemoryStore fixture
cf34afc

docs: release notes
16cb226

test: remove prematurely-added mock s3 fixture
b28eaee

chore: move v3 tests into v3 folder
0215b97

chore: type hints
e33ca6e

test: add schema for group method tests
f237172

chore: add type for zarr_formats
4605446

chore: remove localstore for now
6e5ded9

test: add __init__.py to support imports from top-level conftest.py, …
f81b6f3
…and add some docstrings, and remove redundant def

fix: return valid JSON from GroupMetadata.to_bytes for v2 metadata
c768ca8

fix: don't use a type as a value
c405506

test: add getitem test
5b96554

fix: replace reference to nonexistent method in with , which does exist
d77d55f

test: declare v3ness via directory structure, not test file name
05f89c6

add a docstring to _get, and pass auto_mkdir to _put
b2f9bee

fix: add docstring to LocalStore.get_partial_values; adjust body of L…
88f4c46
…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

test: add tests for localstore init, set, get, get_partial
624ff77

feature(store): make list_* methods async generators
7dff5e5

jhamman commented Apr 10, 2024
View reviewed changes

d-v-b added 5 commits April 11, 2024 11:15

fix: Rename children to members; AsyncGroup.members yields tuples of …
b762fa4
…(name, AsyncArray / AsyncGroup) pairs; Group.members repackages these into a dict.

fix: make Group.members return a tuple of str, Array | Group pairs
5574226

Merge branch 'v3' of github.com:zarr-developers/zarr-python into grou…
bcd5c7d
…p_children_fix

fix: revert changes to synchronization code; this is churn that we ne…
d634cbf
…ed to deal with

Merge branch 'v3' into group_children_fix
9b9d146

d-v-b added 22 commits April 12, 2024 14:35

chore: move v3 tests into v3 folder
d264f71

chore: type hints
0741bed

test: add schema for group method tests
eb8a535

chore: add type for zarr_formats
ee2e233

chore: remove localstore for now
01eec6f

test: add __init__.py to support imports from top-level conftest.py, …
acae77a
…and add some docstrings, and remove redundant def

fix: return valid JSON from GroupMetadata.to_bytes for v2 metadata
ebe1548

fix: don't use a type as a value
3dce5e3

test: add getitem test
7f82fdf

fix: replace reference to nonexistent method in with , which does exist
1655ff8

test: declare v3ness via directory structure, not test file name
e8514b1

add a docstring to _get, and pass auto_mkdir to _put
dacacc8

fix: add docstring to LocalStore.get_partial_values; adjust body of L…
5d2a532
…ocalStore.get_partial_values to properly handle the byte_range parameter of LocalStore.get.

test: add tests for localstore init, set, get, get_partial
06d8b04

fix: remove pre-emptive fetching from group.open
d8749de

Merge branch 'v3_group_tests' of github.com:d-v-b/zarr-python into v3…
a2fe4e0
…_group_tests

fix: use removeprefix (removes a substring) instead of strip (removes…
eed03c8
… any member of a set); comment out / avoid tests that cannot pass right now; don't consider implicit groups for v2; check if prefix is present in storage before opening for Group.getitem

xfail v2 tests that are sure to fail; add delitem tests; partition xf…
75f75b1
…ailing tests into subtests

fix: handle byte_range[0] being None
8a14e3b

fix: adjust test for localstore.get to check that get on nonexistent …
459bb42
…keys returns None; correctly create intermediate directories when preparing test data in test_local_store_get_partial

fix: add zarr_format parameter to array creation routines (which rais…
8ef3fec
…es if zarr_format is not 3), and xfail the tests that will hit this condition. add tests for create_group, create_array, and update_attributes methods of asyncgroup.

test: add group init test
b5a7698

jhamman force-pushed the feature/store-list-methods-async-generators branch from 94b00c7 to 7dff5e5Compare April 19, 2024 03:05

jhammanand others added 2 commits April 18, 2024 20:21

Merge branch 'v3_group_tests' of github.com:d-v-b/zarr-python into fe…
9b2003c
…ature/store-list-methods-async-generators

Update src/zarr/v3/store/memory.py
ccb2dfe

jhamman closed this Apr 19, 2024