Cannot create FSStore for unlistable http URLs

[rabernat]

Minimal, reproducible code sample, a copy-pastable example if possible

import zarr
import fsspec

# public
url = 'https://g-c70289.0da32.08cc.data.globus.org/pyqg_ensemble/001.zarr'

# this errors
store = zarr.storage.FSStore(url)

# because of this
fs = fsspec.implementations.http.HTTPFileSystem()
fs.isdir(url) # -> False

# however, there is consolidated metadata
data = fs.get_mapper(url)['.zmetadata']

Problem description

We often store Zarr on HTTP servers that don't support directory listing. This works because Zarr consolidated metadata tells us everything we need to know about the data available, so no listing is required.

However, Zarr won't let me create such a Store, as shown in the example above.

There needs to be a workaround to allow unlistable http stores with consolidated metadata.

Version and installation information

Please provide the following:

• zarr 2.11.0
• fsspec 2022.02.0

This came up in pangeo-forge/pangeo-forge-recipes#222

[rabernat]

This is the check that triggers the error

zarr-python/zarr/storage.py

Lines 1331 to 1332 in 8d2b651

	if self.fs.exists(self.path) and not self.fs.isdir(self.path):
	raise FSPathExistNotDir(url)

And here is what happens when you try to call open_conslidated on such a store

zarr.open_consolidated(url)

---------------------------------------------------------------------------
FSPathExistNotDir Traceback (most recent call last)
Input In [90], in <cell line: 1>()
----> 1 zarr.open_consolidated(url)

File /opt/miniconda3/envs/rechunker/lib/python3.9/site-packages/zarr/convenience.py:1182, in open_consolidated(store, metadata_key, mode, **kwargs)
 1179 from .storage import ConsolidatedMetadataStore
 1181 # normalize parameters
-> 1182 store = normalize_store_arg(store, storage_options=kwargs.get("storage_options"))
 1183 if mode not in {'r', 'r+'}:
 1184 raise ValueError("invalid mode, expected either 'r' or 'r+'; found {!r}"
 1185 .format(mode))

File /opt/miniconda3/envs/rechunker/lib/python3.9/site-packages/zarr/storage.py:115, in normalize_store_arg(store, clobber, storage_options, mode)
 113 mode = mode if clobber else "r"
 114 if "://" in store or "::" in store:
--> 115 return FSStore(store, mode=mode, **(storage_options or {}))
 116 elif storage_options:
 117 raise ValueError("storage_options passed with non-fsspec path")

File /opt/miniconda3/envs/rechunker/lib/python3.9/site-packages/zarr/storage.py:1142, in FSStore.__init__(self, url, normalize_keys, key_separator, mode, exceptions, dimension_separator, **storage_options)
 1140 self._dimension_separator = dimension_separator
 1141 if self.fs.exists(self.path) and not self.fs.isdir(self.path):
-> 1142 raise FSPathExistNotDir(url)

FSPathExistNotDir: path exists but is not a directory: %r

[rabernat]

@martindurant - any thoughts on how to best address this?

[martindurant]

Hm, yes - for the special case of consolidated, we should just attempt to read the .zmetadata immediately; otherwise a .zarray/.zattrs would be OK too for a bare array, but we would not be able to read any non-consolidated group without ls.

[martindurant]

More info: the http driver concludes that something is/may be a directory, if it includes HTML links to URLs have the prefix of the original URL. However, I think what I wrote above is still correct, we should not need to do isdir at all for this case.

[rabernat]

So what is the path forward here? Can we just remove this check and make FSStore optionally unlistable? Will that break anything?

[martindurant]

Personally, I think it's fine to skip the check and allow any error to happen later upon trying to list a group's members.

However, this is for zarr v2, so we would need a backport?
Alternatively, we could offer an option on the http FS implementation to always give True for isdir(), since any URL might contain more pages in URLs of the same prefix. This would not need a change in zarr.

--- a/fsspec/implementations/http.py
+++ b/fsspec/implementations/http.py
@@ -480,10 +480,7 @@ class HTTPFileSystem(AsyncFileSystem):

 async def _isdir(self, path):
 # override, since all URLs are (also) files
- try:
- return bool(await self._ls(path))
- except (FileNotFoundError, ValueError):
- return False
+ return True