Wrap sync fs for xarray.to_zarr

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

[moradology]

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

[moradology]

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

[jhamman]

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

[moradology]

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2533 +/- ##
==========================================
- Coverage 99.98% 90.27% -9.72% 
==========================================
Files 38 59 +21 Lines 14718 6468 -8250 ==========================================
- Hits 14716 5839 -8877 - Misses 2 629 +627 

Files with missing lines	Coverage Δ
src/zarr/storage/_fsspec.py	84.48% <ø> (ø)

... and 96 files with indirect coverage changes

[martindurant]

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

• I am happy to move the import as requested, to get that done
• I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

[martindurant]

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.