Working with remote data

[LucaMarconato]

Working with remote data is currently only partially supported. Here is a collections of related issues and pull requests that constitute a good point to get started from.

Any user request/feedback from users is highly welcomed! 😊

• Support for consolidated remote zarr #278
• Support remote private storage by consistent use of substore #442
• Support for private remote object storage #441

[vjcitn]

What is meant by "consolidated" here in #278 ?

[LucaMarconato]

Hi @vjcitn, it refers to putting all the metadata of a Zarr store into a single file at the root level. This is done for two reasons: 1) reduce the number of requests to the remote data; 2) actually be able to discover which data is available in a store for those file systems for which ls is not available, such as how it's often the case for S3 object storages.

You can find more details and info in the docs from the Zarr-Python package.

[pakiessling]

@berombau Hi Benjamin, I am trying our your branch. Really cool work.

importzarrfromspatialdataimportSpatialDatas3_address="https://s3.embl.de/spatialdata/spatialdata-sandbox/merfish.zarr/"root=zarr.open_consolidated(s3_address, mode="r", metadata_key="zmetadata")
root.info>... Groupsimages, points, shapes, tables ...
sdata=SpatialData.read(root)
sdata>FileNotFoundError: https://s3.embl.de/spatialdata/spatialdata-sandbox/merfish.zarr/

any idea why this is not working?

[berombau]

Hi @pakiessling ,

Thanks for trying out the branch. Just to be clear, you tried this with the latest support effort located at the draft PR #907? This combines a lot of previous PRs that are yet to be merged and on top of those adds remote support for additional elements.

Can you try just reading from the URL string with SpatialData.read instead of first creating a custom store? It also depends on the storage configuration of the EMBL S3 host and the files accessible there, so maybe also try with a different storage instance of interest to see if that's the cause.

@LucaMarconato

[pakiessling]

Ah I was indeed on the wrong branch. Akward..
For pip install git+https://github.com/berombau/spatialdata@cloud-future I always get some hatchling non-sense:
AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable'

[LucaMarconato]

The most advanced PR for reading data from the cloud is #907, but after adding Zarr v3 support we couldn't merge it right away. This will require more work from our side.