[v3] fix: zarr v2 compatibility fixes for Dask - #2186

Merged
jhamman merged 20 commits into
v3from
fix/dask-compat
Oct 1, 2024
Merged

[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from
fix/dask-compat

Conversation

@jhamman

@jhammanjhamman commented Sep 14, 2024•
edited
Loading

Copy link
Copy Markdown
Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage

closes#1953
xref: dask/dask#11388

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman commented Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/core/array.py Outdated
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment on lines +121 to +125
if all(isinstance(c, (tuple | list)) for c in chunks):
# take first chunk size for each dimension
chunks = (
c[0] for c in chunks
) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])

@jhammanjhammanSep 14, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x

@jhammanjhamman mentioned this pull request Sep 15, 2024
Closed
@will-moore

will-moore commented Sep 16, 2024

Copy link
Copy Markdown
Contributor

Hi @jhamman - thanks for looking into this!
I tried to remember where I got to when I was looking into this before...

Here is what we're doing to load remote data with dask and Zarr v2...

# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...

import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?

Many thanks,
Will

This was referenced Sep 17, 2024
Closed
Closed
@TomAugspurger

TomAugspurger commented Sep 17, 2024

Copy link
Copy Markdown
Contributor

Thanks for trying this out @will-moore. Can you share the full traceback?

In the meantime, I would expect something like

g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")

Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.

jhamman
jhamman commented Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
@jhamman
jhamman requested a review from d-v-bSeptember 17, 2024 15:46
@will-moore

will-moore commented Sep 17, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!
So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:

store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:

zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:

this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!
Might have to re-install anaconda, or try some other option...

Thanks for your help!

@will-moorewill-moore mentioned this pull request Sep 17, 2024
Closed
@jhammanjhamman mentioned this pull request Sep 18, 2024
Closed
4 tasks
@will-moorewill-moore mentioned this pull request Sep 19, 2024
Closed
@jrbourbeaujrbourbeau mentioned this pull request Sep 20, 2024
Merged
3 tasks
@TomAugspurgerTomAugspurger mentioned this pull request Sep 25, 2024
Merged
6 tasks
jhamman
jhamman commented Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/core/array.py Outdated
Comment threadsrc/zarr/core/array.py
Comment on lines +2394 to +2399
parents = [
AsyncGroup(
metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
store_path=StorePath(store=node.store_path.store, path=""),
)
]

@jhammanjhammanSep 29, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.

d-v-b
d-v-b requested changes Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/api/asynchronous.py Outdated
Comment threadsrc/zarr/core/array.py Outdated
Comment threadsrc/zarr/core/array.py Outdated
jhamman
jhamman commented Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py Outdated
d-v-b
d-v-b approved these changes Oct 1, 2024
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think

@jhammanjhamman mentioned this pull request Oct 1, 2024
Merged
6 tasks
TomAugspurger
TomAugspurger approved these changes Oct 1, 2024
View reviewed changes

@TomAugspurgerTomAugspurger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.

Comment threadsrc/zarr/core/array.py Outdated
# To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
parents = _build_parents(self)

logger.debug("Ensure parents: %s", parents)

@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?

Comment threadsrc/zarr/core/array.py
parents = []
store = node.store_path.store
path = node.store_path.path
if not path:

@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.

@jhammanjhammanOct 1, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.

jhamman
jhamman commented Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py Outdated
@jhamman
jhamman merged commit f3a2e0a into v3Oct 1, 2024
@dstansby
dstansby deleted the fix/dask-compat branch October 13, 2024 07:31
@dstansby
dstansby restored the fix/dask-compat branch October 13, 2024 07:31
@jhamman
jhamman deleted the fix/dask-compat branch October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
[v3] fix: zarr v2 compatibility fixes for Dask - #2186
Merged
jhamman merged 20 commits into
v3from 
fix/dask-compat
Oct 1, 2024
Merged
[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from 
fix/dask-compat
Conversation
@jhamman
@jhammanjhamman
commented
Sep 14, 2024•

edited

Loading

Copy link


Copy Markdown

Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage
closes#1953

xref: dask/dask#11388
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman
commented
Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py

Comment on lines

+121
 to 
+125

 if all(isinstance(c, (tuple | list)) for c in chunks):
 # take first chunk size for each dimension
 chunks = (
 c[0] for c in chunks
 ) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
@jhammanjhammanSep 14, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x
@jhammanjhamman
mentioned this pull request
Sep 15, 2024

Closed

@will-moore
will-moore
commented
Sep 16, 2024

Copy link


Copy Markdown

Contributor
Hi @jhamman - thanks for looking into this!

I tried to remember where I got to when I was looking into this before...
Here is what we're doing to load remote data with dask and Zarr v2...
# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...
import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?
Many thanks,

Will

This was referenced Sep 17, 2024
 Closed

 Closed

@TomAugspurger
TomAugspurger
commented
Sep 17, 2024

Copy link


Copy Markdown

Contributor
Thanks for trying this out @will-moore. Can you share the full traceback?
In the meantime, I would expect something like
g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")
Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.
jhamman
jhamman
commented
Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

@jhamman
jhamman
requested a review
from d-v-bSeptember 17, 2024 15:46
@will-moore
will-moore
commented
Sep 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!

So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:
store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:
zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:
this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!

Might have to re-install anaconda, or try some other option...
Thanks for your help!
@will-moorewill-moore
mentioned this pull request
Sep 17, 2024

Closed

@jhammanjhamman
mentioned this pull request
Sep 18, 2024

Closed

4 tasks
@will-moorewill-moore
mentioned this pull request
Sep 19, 2024
 Closed

@jrbourbeaujrbourbeau
mentioned this pull request
Sep 20, 2024
 Merged

3 tasks
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 25, 2024
 Merged

6 tasks
jhamman
jhamman
commented
Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py

Comment on lines

+2394
 to 
+2399

 parents = [
 AsyncGroup(
 metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
 store_path=StorePath(store=node.store_path.store, path=""),
 )
 ]
@jhammanjhammanSep 29, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.
d-v-b
d-v-b
requested changes
Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

jhamman
jhamman
commented
Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

d-v-b
d-v-b
approved these changes
Oct 1, 2024
View reviewed changes
@d-v-bd-v-b
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think
@jhammanjhamman
mentioned this pull request
Oct 1, 2024
 Merged

6 tasks
TomAugspurger
TomAugspurger
approved these changes
Oct 1, 2024
View reviewed changes
@TomAugspurgerTomAugspurger
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.
Comment threadsrc/zarr/core/array.py
Outdated

 # To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
 parents = _build_parents(self)

 logger.debug("Ensure parents: %s", parents)
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?
Comment threadsrc/zarr/core/array.py
 parents = []
 store = node.store_path.store
 path = node.store_path.path
 if not path:
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.
@jhammanjhammanOct 1, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.
jhamman
jhamman
commented
Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

@jhamman
jhamman
merged commit f3a2e0a
into
v3Oct 1, 2024
@dstansby
dstansby
deleted the

fix/dask-compat

branch
October 13, 2024 07:31
@dstansby
dstansby
restored the

fix/dask-compat

branch
October 13, 2024 07:31
@jhamman
jhamman
deleted the

fix/dask-compat

branch
October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility 

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[v3] fix: zarr v2 compatibility fixes for Dask - #2186
Merged
jhamman merged 20 commits into
v3from 
fix/dask-compat
Oct 1, 2024
Merged
[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from 
fix/dask-compat
Conversation
@jhamman
@jhammanjhamman
commented
Sep 14, 2024•

edited

Loading

Copy link


Copy Markdown

Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage
closes#1953

xref: dask/dask#11388
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman
commented
Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py

Comment on lines

+121
 to 
+125

 if all(isinstance(c, (tuple | list)) for c in chunks):
 # take first chunk size for each dimension
 chunks = (
 c[0] for c in chunks
 ) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
@jhammanjhammanSep 14, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x
@jhammanjhamman
mentioned this pull request
Sep 15, 2024

Closed

@will-moore
will-moore
commented
Sep 16, 2024

Copy link


Copy Markdown

Contributor
Hi @jhamman - thanks for looking into this!

I tried to remember where I got to when I was looking into this before...
Here is what we're doing to load remote data with dask and Zarr v2...
# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...
import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?
Many thanks,

Will

This was referenced Sep 17, 2024
 Closed

 Closed

@TomAugspurger
TomAugspurger
commented
Sep 17, 2024

Copy link


Copy Markdown

Contributor
Thanks for trying this out @will-moore. Can you share the full traceback?
In the meantime, I would expect something like
g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")
Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.
jhamman
jhamman
commented
Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

@jhamman
jhamman
requested a review
from d-v-bSeptember 17, 2024 15:46
@will-moore
will-moore
commented
Sep 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!

So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:
store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:
zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:
this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!

Might have to re-install anaconda, or try some other option...
Thanks for your help!
@will-moorewill-moore
mentioned this pull request
Sep 17, 2024

Closed

@jhammanjhamman
mentioned this pull request
Sep 18, 2024

Closed

4 tasks
@will-moorewill-moore
mentioned this pull request
Sep 19, 2024
 Closed

@jrbourbeaujrbourbeau
mentioned this pull request
Sep 20, 2024
 Merged

3 tasks
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 25, 2024
 Merged

6 tasks
jhamman
jhamman
commented
Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py

Comment on lines

+2394
 to 
+2399

 parents = [
 AsyncGroup(
 metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
 store_path=StorePath(store=node.store_path.store, path=""),
 )
 ]
@jhammanjhammanSep 29, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.
d-v-b
d-v-b
requested changes
Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

jhamman
jhamman
commented
Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

d-v-b
d-v-b
approved these changes
Oct 1, 2024
View reviewed changes
@d-v-bd-v-b
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think
@jhammanjhamman
mentioned this pull request
Oct 1, 2024
 Merged

6 tasks
TomAugspurger
TomAugspurger
approved these changes
Oct 1, 2024
View reviewed changes
@TomAugspurgerTomAugspurger
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.
Comment threadsrc/zarr/core/array.py
Outdated

 # To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
 parents = _build_parents(self)

 logger.debug("Ensure parents: %s", parents)
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?
Comment threadsrc/zarr/core/array.py
 parents = []
 store = node.store_path.store
 path = node.store_path.path
 if not path:
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.
@jhammanjhammanOct 1, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.
jhamman
jhamman
commented
Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

@jhamman
jhamman
merged commit f3a2e0a
into
v3Oct 1, 2024
@dstansby
dstansby
deleted the

fix/dask-compat

branch
October 13, 2024 07:31
@dstansby
dstansby
restored the

fix/dask-compat

branch
October 13, 2024 07:31
@jhamman
jhamman
deleted the

fix/dask-compat

branch
October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility 

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[v3] fix: zarr v2 compatibility fixes for Dask - #2186
Merged
jhamman merged 20 commits into
v3from 
fix/dask-compat
Oct 1, 2024
Merged
[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from 
fix/dask-compat
Conversation
@jhamman
@jhammanjhamman
commented
Sep 14, 2024•

edited

Loading

Copy link


Copy Markdown

Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage
closes#1953

xref: dask/dask#11388
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman
commented
Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py

Comment on lines

+121
 to 
+125

 if all(isinstance(c, (tuple | list)) for c in chunks):
 # take first chunk size for each dimension
 chunks = (
 c[0] for c in chunks
 ) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
@jhammanjhammanSep 14, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x
@jhammanjhamman
mentioned this pull request
Sep 15, 2024

Closed

@will-moore
will-moore
commented
Sep 16, 2024

Copy link


Copy Markdown

Contributor
Hi @jhamman - thanks for looking into this!

I tried to remember where I got to when I was looking into this before...
Here is what we're doing to load remote data with dask and Zarr v2...
# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...
import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?
Many thanks,

Will

This was referenced Sep 17, 2024
 Closed

 Closed

@TomAugspurger
TomAugspurger
commented
Sep 17, 2024

Copy link


Copy Markdown

Contributor
Thanks for trying this out @will-moore. Can you share the full traceback?
In the meantime, I would expect something like
g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")
Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.
jhamman
jhamman
commented
Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

@jhamman
jhamman
requested a review
from d-v-bSeptember 17, 2024 15:46
@will-moore
will-moore
commented
Sep 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!

So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:
store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:
zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:
this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!

Might have to re-install anaconda, or try some other option...
Thanks for your help!
@will-moorewill-moore
mentioned this pull request
Sep 17, 2024

Closed

@jhammanjhamman
mentioned this pull request
Sep 18, 2024

Closed

4 tasks
@will-moorewill-moore
mentioned this pull request
Sep 19, 2024
 Closed

@jrbourbeaujrbourbeau
mentioned this pull request
Sep 20, 2024
 Merged

3 tasks
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 25, 2024
 Merged

6 tasks
jhamman
jhamman
commented
Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py

Comment on lines

+2394
 to 
+2399

 parents = [
 AsyncGroup(
 metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
 store_path=StorePath(store=node.store_path.store, path=""),
 )
 ]
@jhammanjhammanSep 29, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.
d-v-b
d-v-b
requested changes
Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

jhamman
jhamman
commented
Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

d-v-b
d-v-b
approved these changes
Oct 1, 2024
View reviewed changes
@d-v-bd-v-b
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think
@jhammanjhamman
mentioned this pull request
Oct 1, 2024
 Merged

6 tasks
TomAugspurger
TomAugspurger
approved these changes
Oct 1, 2024
View reviewed changes
@TomAugspurgerTomAugspurger
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.
Comment threadsrc/zarr/core/array.py
Outdated

 # To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
 parents = _build_parents(self)

 logger.debug("Ensure parents: %s", parents)
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?
Comment threadsrc/zarr/core/array.py
 parents = []
 store = node.store_path.store
 path = node.store_path.path
 if not path:
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.
@jhammanjhammanOct 1, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.
jhamman
jhamman
commented
Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

@jhamman
jhamman
merged commit f3a2e0a
into
v3Oct 1, 2024
@dstansby
dstansby
deleted the

fix/dask-compat

branch
October 13, 2024 07:31
@dstansby
dstansby
restored the

fix/dask-compat

branch
October 13, 2024 07:31
@jhamman
jhamman
deleted the

fix/dask-compat

branch
October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility 

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
[v3] fix: zarr v2 compatibility fixes for Dask - #2186
Merged
jhamman merged 20 commits into
v3from 
fix/dask-compat
Oct 1, 2024
Merged
[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from 
fix/dask-compat
Conversation
@jhamman
@jhammanjhamman
commented
Sep 14, 2024•

edited

Loading

Copy link


Copy Markdown

Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage
closes#1953

xref: dask/dask#11388
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman
commented
Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py

Comment on lines

+121
 to 
+125

 if all(isinstance(c, (tuple | list)) for c in chunks):
 # take first chunk size for each dimension
 chunks = (
 c[0] for c in chunks
 ) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
@jhammanjhammanSep 14, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x
@jhammanjhamman
mentioned this pull request
Sep 15, 2024

Closed

@will-moore
will-moore
commented
Sep 16, 2024

Copy link


Copy Markdown

Contributor
Hi @jhamman - thanks for looking into this!

I tried to remember where I got to when I was looking into this before...
Here is what we're doing to load remote data with dask and Zarr v2...
# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...
import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?
Many thanks,

Will

This was referenced Sep 17, 2024
 Closed

 Closed

@TomAugspurger
TomAugspurger
commented
Sep 17, 2024

Copy link


Copy Markdown

Contributor
Thanks for trying this out @will-moore. Can you share the full traceback?
In the meantime, I would expect something like
g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")
Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.
jhamman
jhamman
commented
Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

@jhamman
jhamman
requested a review
from d-v-bSeptember 17, 2024 15:46
@will-moore
will-moore
commented
Sep 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!

So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:
store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:
zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:
this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!

Might have to re-install anaconda, or try some other option...
Thanks for your help!
@will-moorewill-moore
mentioned this pull request
Sep 17, 2024

Closed

@jhammanjhamman
mentioned this pull request
Sep 18, 2024

Closed

4 tasks
@will-moorewill-moore
mentioned this pull request
Sep 19, 2024
 Closed

@jrbourbeaujrbourbeau
mentioned this pull request
Sep 20, 2024
 Merged

3 tasks
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 25, 2024
 Merged

6 tasks
jhamman
jhamman
commented
Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py

Comment on lines

+2394
 to 
+2399

 parents = [
 AsyncGroup(
 metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
 store_path=StorePath(store=node.store_path.store, path=""),
 )
 ]
@jhammanjhammanSep 29, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.
d-v-b
d-v-b
requested changes
Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

jhamman
jhamman
commented
Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

d-v-b
d-v-b
approved these changes
Oct 1, 2024
View reviewed changes
@d-v-bd-v-b
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think
@jhammanjhamman
mentioned this pull request
Oct 1, 2024
 Merged

6 tasks
TomAugspurger
TomAugspurger
approved these changes
Oct 1, 2024
View reviewed changes
@TomAugspurgerTomAugspurger
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.
Comment threadsrc/zarr/core/array.py
Outdated

 # To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
 parents = _build_parents(self)

 logger.debug("Ensure parents: %s", parents)
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?
Comment threadsrc/zarr/core/array.py
 parents = []
 store = node.store_path.store
 path = node.store_path.path
 if not path:
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.
@jhammanjhammanOct 1, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.
jhamman
jhamman
commented
Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

@jhamman
jhamman
merged commit f3a2e0a
into
v3Oct 1, 2024
@dstansby
dstansby
deleted the

fix/dask-compat

branch
October 13, 2024 07:31
@dstansby
dstansby
restored the

fix/dask-compat

branch
October 13, 2024 07:31
@jhamman
jhamman
deleted the

fix/dask-compat

branch
October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility 

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[v3] fix: zarr v2 compatibility fixes for Dask - #2186
Merged
jhamman merged 20 commits into
v3from 
fix/dask-compat
Oct 1, 2024
Merged
[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from 
fix/dask-compat
Conversation
@jhamman
@jhammanjhamman
commented
Sep 14, 2024•

edited

Loading

Copy link


Copy Markdown

Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage
closes#1953

xref: dask/dask#11388
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman
commented
Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py

Comment on lines

+121
 to 
+125

 if all(isinstance(c, (tuple | list)) for c in chunks):
 # take first chunk size for each dimension
 chunks = (
 c[0] for c in chunks
 ) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
@jhammanjhammanSep 14, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x
@jhammanjhamman
mentioned this pull request
Sep 15, 2024

Closed

@will-moore
will-moore
commented
Sep 16, 2024

Copy link


Copy Markdown

Contributor
Hi @jhamman - thanks for looking into this!

I tried to remember where I got to when I was looking into this before...
Here is what we're doing to load remote data with dask and Zarr v2...
# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...
import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?
Many thanks,

Will

This was referenced Sep 17, 2024
 Closed

 Closed

@TomAugspurger
TomAugspurger
commented
Sep 17, 2024

Copy link


Copy Markdown

Contributor
Thanks for trying this out @will-moore. Can you share the full traceback?
In the meantime, I would expect something like
g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")
Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.
jhamman
jhamman
commented
Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

@jhamman
jhamman
requested a review
from d-v-bSeptember 17, 2024 15:46
@will-moore
will-moore
commented
Sep 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!

So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:
store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:
zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:
this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!

Might have to re-install anaconda, or try some other option...
Thanks for your help!
@will-moorewill-moore
mentioned this pull request
Sep 17, 2024

Closed

@jhammanjhamman
mentioned this pull request
Sep 18, 2024

Closed

4 tasks
@will-moorewill-moore
mentioned this pull request
Sep 19, 2024
 Closed

@jrbourbeaujrbourbeau
mentioned this pull request
Sep 20, 2024
 Merged

3 tasks
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 25, 2024
 Merged

6 tasks
jhamman
jhamman
commented
Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py

Comment on lines

+2394
 to 
+2399

 parents = [
 AsyncGroup(
 metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
 store_path=StorePath(store=node.store_path.store, path=""),
 )
 ]
@jhammanjhammanSep 29, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.
d-v-b
d-v-b
requested changes
Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

jhamman
jhamman
commented
Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

d-v-b
d-v-b
approved these changes
Oct 1, 2024
View reviewed changes
@d-v-bd-v-b
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think
@jhammanjhamman
mentioned this pull request
Oct 1, 2024
 Merged

6 tasks
TomAugspurger
TomAugspurger
approved these changes
Oct 1, 2024
View reviewed changes
@TomAugspurgerTomAugspurger
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.
Comment threadsrc/zarr/core/array.py
Outdated

 # To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
 parents = _build_parents(self)

 logger.debug("Ensure parents: %s", parents)
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?
Comment threadsrc/zarr/core/array.py
 parents = []
 store = node.store_path.store
 path = node.store_path.path
 if not path:
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.
@jhammanjhammanOct 1, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.
jhamman
jhamman
commented
Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

@jhamman
jhamman
merged commit f3a2e0a
into
v3Oct 1, 2024
@dstansby
dstansby
deleted the

fix/dask-compat

branch
October 13, 2024 07:31
@dstansby
dstansby
restored the

fix/dask-compat

branch
October 13, 2024 07:31
@jhamman
jhamman
deleted the

fix/dask-compat

branch
October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility 

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[v3] fix: zarr v2 compatibility fixes for Dask - #2186
Merged
jhamman merged 20 commits into
v3from 
fix/dask-compat
Oct 1, 2024
Merged
[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from 
fix/dask-compat
Conversation
@jhamman
@jhammanjhamman
commented
Sep 14, 2024•

edited

Loading

Copy link


Copy Markdown

Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage
closes#1953

xref: dask/dask#11388
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman
commented
Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py

Comment on lines

+121
 to 
+125

 if all(isinstance(c, (tuple | list)) for c in chunks):
 # take first chunk size for each dimension
 chunks = (
 c[0] for c in chunks
 ) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
@jhammanjhammanSep 14, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x
@jhammanjhamman
mentioned this pull request
Sep 15, 2024

Closed

@will-moore
will-moore
commented
Sep 16, 2024

Copy link


Copy Markdown

Contributor
Hi @jhamman - thanks for looking into this!

I tried to remember where I got to when I was looking into this before...
Here is what we're doing to load remote data with dask and Zarr v2...
# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...
import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?
Many thanks,

Will

This was referenced Sep 17, 2024
 Closed

 Closed

@TomAugspurger
TomAugspurger
commented
Sep 17, 2024

Copy link


Copy Markdown

Contributor
Thanks for trying this out @will-moore. Can you share the full traceback?
In the meantime, I would expect something like
g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")
Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.
jhamman
jhamman
commented
Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

@jhamman
jhamman
requested a review
from d-v-bSeptember 17, 2024 15:46
@will-moore
will-moore
commented
Sep 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!

So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:
store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:
zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:
this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!

Might have to re-install anaconda, or try some other option...
Thanks for your help!
@will-moorewill-moore
mentioned this pull request
Sep 17, 2024

Closed

@jhammanjhamman
mentioned this pull request
Sep 18, 2024

Closed

4 tasks
@will-moorewill-moore
mentioned this pull request
Sep 19, 2024
 Closed

@jrbourbeaujrbourbeau
mentioned this pull request
Sep 20, 2024
 Merged

3 tasks
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 25, 2024
 Merged

6 tasks
jhamman
jhamman
commented
Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py

Comment on lines

+2394
 to 
+2399

 parents = [
 AsyncGroup(
 metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
 store_path=StorePath(store=node.store_path.store, path=""),
 )
 ]
@jhammanjhammanSep 29, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.
d-v-b
d-v-b
requested changes
Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

jhamman
jhamman
commented
Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

d-v-b
d-v-b
approved these changes
Oct 1, 2024
View reviewed changes
@d-v-bd-v-b
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think
@jhammanjhamman
mentioned this pull request
Oct 1, 2024
 Merged

6 tasks
TomAugspurger
TomAugspurger
approved these changes
Oct 1, 2024
View reviewed changes
@TomAugspurgerTomAugspurger
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.
Comment threadsrc/zarr/core/array.py
Outdated

 # To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
 parents = _build_parents(self)

 logger.debug("Ensure parents: %s", parents)
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?
Comment threadsrc/zarr/core/array.py
 parents = []
 store = node.store_path.store
 path = node.store_path.path
 if not path:
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.
@jhammanjhammanOct 1, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.
jhamman
jhamman
commented
Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

@jhamman
jhamman
merged commit f3a2e0a
into
v3Oct 1, 2024
@dstansby
dstansby
deleted the

fix/dask-compat

branch
October 13, 2024 07:31
@dstansby
dstansby
restored the

fix/dask-compat

branch
October 13, 2024 07:31
@jhamman
jhamman
deleted the

fix/dask-compat

branch
October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility 

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
[v3] fix: zarr v2 compatibility fixes for Dask - #2186
Merged
jhamman merged 20 commits into
v3from 
fix/dask-compat
Oct 1, 2024
Merged
[v3] fix: zarr v2 compatibility fixes for Dask#2186
jhamman merged 20 commits into
v3from 
fix/dask-compat
Conversation
@jhamman
@jhammanjhamman
commented
Sep 14, 2024•

edited

Loading

Copy link


Copy Markdown

Member
  • port normalize_chunks from v2
  • add array.store property
  • default to append in create
  • rename zarr.store to zarr.storage
closes#1953

xref: dask/dask#11388
TODO:
  •  Add unit tests and/or doctests in docstrings
  •  Add docstrings and API docs for any new/modified user-facing classes and functions
  •  New/modified features documented in docs/tutorial.rst
  •  Changes documented in docs/release.rst
  •  GitHub Actions have all passed
  •  Test coverage is 100% (Codecov passes)
@jhamman
fix: zarr v2 compatability fixes
fe49f5f
- port normalize_chunks from v2
- add array.store property
- default to append in create
jhamman
jhamman
commented
Sep 14, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/core/chunk_grids.py

Comment on lines

+121
 to 
+125

 if all(isinstance(c, (tuple | list)) for c in chunks):
 # take first chunk size for each dimension
 chunks = (
 c[0] for c in chunks
 ) # TODO: check/error/warn for irregular chunks (e.g. if c[0] != c[1:-1])
@jhammanjhammanSep 14, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

except for this block, the rest of this function came from 2.x
@jhammanjhamman
mentioned this pull request
Sep 15, 2024

Closed

@will-moore
will-moore
commented
Sep 16, 2024

Copy link


Copy Markdown

Contributor
Hi @jhamman - thanks for looking into this!

I tried to remember where I got to when I was looking into this before...
Here is what we're doing to load remote data with dask and Zarr v2...
# dask==2023.5.0
# dask-image==2022.9.0
# zarr==2.18.2
import dask.array as da
from zarr import Array, Group, open
from zarr.storage import FSStore
import json
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr"
store = FSStore(url)
zattrs = json.loads(store.get(".zattrs"))
array_path = zattrs["multiscales"][0]["datasets"][0]["path"]
array_data = da.from_zarr(store, array_path)
print(array_data)
# dask.array<from-zarr, shape=(2, 236, 275, 271), dtype=uint16, chunksize=(1, 1, 275, 271), chunktype=numpy.ndarray>

With zarr v3 I'm not getting as far as opening with dask as I'm failing to access any remote data, using a couple of different approaches here (using zarr from this branch), both failing with exceptions noted below...
import zarr
from zarr.store import RemoteStore
import s3fs
url = "https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr"
store = RemoteStore(url)
# RuntimeError: Timeout context manager should be used inside a task
g = zarr.group(store = store)
s3 = s3fs.S3FileSystem(anon=True, client_kwargs={'endpoint_url': 'https://uk1s3.embassy.ebi.ac.uk'})
store = s3fs.S3Map(root='idr/share/ome2024-ngff-challenge/0.0.5/6001240.zarr', s3=s3, check=False)
# TypeError - only LocalStore or MemoryStore is supported
g = zarr.group(store = store)

Is there some other way that I can try to load remote data?
Many thanks,

Will

This was referenced Sep 17, 2024
 Closed

 Closed

@TomAugspurger
TomAugspurger
commented
Sep 17, 2024

Copy link


Copy Markdown

Contributor
Thanks for trying this out @will-moore. Can you share the full traceback?
In the meantime, I would expect something like
g=zarr.open_group(store=zarr.store.RemoteStore("s3://idr", endpoint_url="http://uk1s3.embassy.ebi.ac.uk", anon=True), path="zarr/v0.4/idr0062A/6001240.zarr")
Can you let us know if that works? It doesn't for me, but I'm not sure if some network / authentication setup is required.
jhamman
jhamman
commented
Sep 17, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

@jhamman
jhamman
requested a review
from d-v-bSeptember 17, 2024 15:46
@will-moore
will-moore
commented
Sep 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
That sample you tried there is zarr v2 data, and it also seemed to need https rather than http!

So this works for me on another machine, with zarr==3.0.0a0, reading both zarr v2 and v3 data:
store=zarr.store.RemoteStore("s3://idr", endpoint_url="https://uk1s3.embassy.ebi.ac.uk", anon=True)
g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
print('zarr v3 group', g)
g2 = zarr.open_group(store=store, path="zarr/v0.4/idr0062A/6001240.zarr", zarr_version=2)
print('zarr v2', g2)

gives me:
zarr v3 group Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/share/ome2024-ngff-challenge/idr0048/9846151.zarr>)
zarr v2 Group(_async_group=<AsyncGroup Remote fsspec store: S3FileSystem , idr/zarr/v0.4/idr0062A/6001240.zarr>)

On my machine, I unfortunately get:
this error
 g = zarr.open_group(store=store, path="share/ome2024-ngff-challenge/idr0048/9846151.zarr")
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/synchronous.py", line 175, in open_group
sync(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 92, in sync
raise return_result
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/sync.py", line 51, in _runner
return await coro
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/api/asynchronous.py", line 523, in open_group
return await AsyncGroup.open(store_path, zarr_format=zarr_format)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/group.py", line 150, in open
zarr_json_bytes = await (store_path / ZARR_JSON).get()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/core.py", line 35, in get
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/zarr/store/remote.py", line 98, in get
await (
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1128, in _cat_file
return await _error_wrapper(_call_and_read, retries=self.retries)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 145, in _error_wrapper
raise err
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
return await func(*args, **kwargs)
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 1115, in _call_and_read
resp = await self._call_s3(
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 358, in _call_s3
await self.set_session()
File "/Users/wmoore/opt/anaconda3/envs/zarr_v3/lib/python3.10/site-packages/s3fs/core.py", line 519, in set_session
self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: AioSession.__init__() got an unexpected keyword argument '//'

Which is an issue that seems to affect just me, as I already reported at ome/ome2024-ngff-challenge#22 and I can't seem to fix on my machine, which is kinda annoying!

Might have to re-install anaconda, or try some other option...
Thanks for your help!
@will-moorewill-moore
mentioned this pull request
Sep 17, 2024

Closed

@jhammanjhamman
mentioned this pull request
Sep 18, 2024

Closed

4 tasks
@will-moorewill-moore
mentioned this pull request
Sep 19, 2024
 Closed

@jrbourbeaujrbourbeau
mentioned this pull request
Sep 20, 2024
 Merged

3 tasks
@TomAugspurgerTomAugspurger
mentioned this pull request
Sep 25, 2024
 Merged

6 tasks
jhamman
jhamman
commented
Sep 29, 2024
View reviewed changes
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py

Comment on lines

+2394
 to 
+2399

 parents = [
 AsyncGroup(
 metadata=GroupMetadata(zarr_format=node.metadata.zarr_format),
 store_path=StorePath(store=node.store_path.store, path=""),
 )
 ]
@jhammanjhammanSep 29, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger - this was needed to enable zarr.create(store={}, path="a") such that the root zarr.json is created in addition to a/zarr.json.
d-v-b
d-v-b
requested changes
Sep 30, 2024
View reviewed changes
Comment threadtests/v3/test_codecs/test_codecs.py
Comment threadsrc/zarr/core/chunk_grids.py
Comment threadsrc/zarr/core/array.py
Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/api/asynchronous.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

Comment threadsrc/zarr/core/array.py
Outdated

jhamman
jhamman
commented
Sep 30, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

d-v-b
d-v-b
approved these changes
Oct 1, 2024
View reviewed changes
@d-v-bd-v-b
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to go i think
@jhammanjhamman
mentioned this pull request
Oct 1, 2024
 Merged

6 tasks
TomAugspurger
TomAugspurger
approved these changes
Oct 1, 2024
View reviewed changes
@TomAugspurgerTomAugspurger
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple minor things.
Comment threadsrc/zarr/core/array.py
Outdated

 # To enable zarr.create(store, path="a/b/c"), we need to create all the intermediate groups.
 parents = _build_parents(self)

 logger.debug("Ensure parents: %s", parents)
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to keep this?
Comment threadsrc/zarr/core/array.py
 parents = []
 store = node.store_path.store
 path = node.store_path.path
 if not path:
@TomAugspurgerTomAugspurgerOct 1, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know whether this branch is covered by an existing test? If not, it might be good to add one.
@jhammanjhammanOct 1, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this path is highly exercised.
jhamman
jhamman
commented
Oct 1, 2024
View reviewed changes
Comment threadsrc/zarr/core/array.py
Outdated

@jhamman
jhamman
merged commit f3a2e0a
into
v3Oct 1, 2024
@dstansby
dstansby
deleted the

fix/dask-compat

branch
October 13, 2024 07:31
@dstansby
dstansby
restored the

fix/dask-compat

branch
October 13, 2024 07:31
@jhamman
jhamman
deleted the

fix/dask-compat

branch
October 15, 2024 12:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TomAugspurgerTomAugspurgerTomAugspurger approved these changes
@d-v-bd-v-bd-v-b approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[v3] Array constructor API compatibility 

4 participants

@jhamman@will-moore@TomAugspurger@d-v-b