Set the cloud storage credentials when opening a datatree from ZarrStore (#9197)

[vlevasseur073]

• ClosesOpening a datatree from S3 bucket with zarr store #9197
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[welcome]

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

[shoyer]

This look good, but I'm concerned that it might break again in the future if we don't add any tests.

Is there any way we could test that storage_options gets passed on properly to Zarr? Maybe with a local directory store?

[vlevasseur073]

Hi Stephan,
Indeed I am not really sure how to efficiently set tests on this. It should probably rather focus on the ZarrStore rather than this backend higher-level API, since here we are only passing this kwarg storage_options to the store. What is always a bit difficult is to get the correct structure of this storage_options to correctly pass the credentials. This could be very carefully documented, but it depends on the backend.
Another possibility could be to have more like an integration test where we simply try to open a sample zarr product from a private cloud storage. This requires to have such a storage and to add the credentials in the CI environment variables.
Please tell me if this helps, or if I can investigate anything.

[TomNicholas]

Is this made redundant by all the extra kwargs that were passed through by #9377? That PR also passes through storage_options.

[TomNicholas]

Actually this PR is still needed because

xarray/xarray/backends/zarr.py

Line 1259 in 19a0428

ds=open_dataset(filename_or_obj, group=parent, engine="zarr", **kwargs)

doesn't pass through storage_options.

The fix here ideally would refactor to use _iter_groups to build a full dict to pass to DataTree.from_dict.

[TomNicholas]

@vlevasseur073 please try this with main - I'm 80% sure we have already implemented what you need!