Basic working FsspecStore

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al

[martindurant]

This works.

• We don't have any list methods, do we need them?
• We don't have a bulk delete
• No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

Fixes#1757

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[martindurant]

@jhamman@d-v-b , for discussion

[pep8speaks]

Hello @martindurant! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-04-14 01:40:25 UTC

[d-v-b]

thanks for this @martindurant, I will see if I have time to play with this locally.

No exception handling is done here, which has been a thorny issue. Shall we do the same as in v2 with expected exceptions -> KeyError? I don't see in Store's API what the expectation is.

I don't think we have expectations at this point. I will try to get a distillation from the v2 issues around this topic. What do you think we should do here?

[martindurant]

I see we have no storage tests, so don't know how to push this any further.

[d-v-b]

I will get some tests for you shortly

[jhamman]

@martindurant - the test suite is in a better place now. Are you up for picking this up?

[martindurant]

For testing ... I could set up a mock s3 or gcs with some pain and CI overhead. Also, I could wrap the fsspec memoryFS in async stuff, but that would not really be testing the async-ness and depends no me doing that correctly. None of the other stores are actually async, right?

[jhamman]

@martindurant - which fsspec implementations have an async-api available?

For now, a mocked s3 backend seems like the way to go.

[martindurant]

which fsspec implementations have an async-api

I think this is a complete list

• s3
• gcs
• azure (both blob and datalake2)
• http
• sshfs (not the builtin ssh/sftp)
• anaconda (released version is still sync)

The following can pass through async, if they wrap an async FS:

• generic
• reference
• dir/prefix

[jhamman]

Got it. Makes sense. For now, let's target a mocked test against s3. We can add a few others as we approach a full release.

[martindurant]

a mocked test against s3

Well, I would use the moto server as s3fs does, which is a real S3 implementation with most of the functionality of the real one (minus permissions, object lifetimes and other unimportant things).

[jhamman]

@martindurant -- this is coming together. I took it for a spin today and ended up needing to make a few changes (suggestions below). But it works!

[martindurant]

I merged from v3 and the suggestions above. I cannot get mypy to play.

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

[jhamman]

@martindurant -- thanks for pushing this forward...

I merged from v3 and the suggestions above. I cannot get mypy to play.

I'm hoping @dstansby can take a look... we can find a way!

By the way, I would suggest that memoryview() and bytes() really ought to work as expected for a Buffer.

cc @madsbk, @akshaysubr, @normanrz

[martindurant]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

[d-v-b]

OK, so the problem with the tests, is that they call get/set in blocking sync code, but the store implementation works in async mode. The test instance is created in sync mode, and gets put on a dedicated fsspec IO thread; but then the async store on the main thread calls the same instance in async mode. Note that moto3 is also running on a thread, to complicate things. I am looking at it.

That's great insight, thank you. Please let me know if there's anything about the StoreTests design that we should change to make this process simpler.

[jhamman]

I see ✅!

Thanks @martindurant for working out the kinks here! (and @d-v-b and @dstansby for chipping in)

[martindurant]

Highlighting this line, which works around tests passing offset-length for data which is b"". Python allows you to slice bytes like that (b""[1:2] == b""), but remote stores do not allow this.

[d-v-b]

I count this as additional evidence that we should switch to start/end semantics for the rest of the stores

[martindurant]

Kerchunk is the exception, storing start/length, mostly because length is generally smaller for chunks in big files.

[d-v-b]

flagging this as s3 abstraction leakage that we might want to address later on by making an s3-specific storage class

[d-v-b]

i think returning the exceptions is the right thing here

[d-v-b]

what's the advantage of going with POSIX semantics here?

[martindurant]

fsspec tries to adhere to posix as much as possible. If we want to exclude the [file] case, we'd have to code that special case into our store.

[d-v-b]

@martindurant could you explain what's happening in this test fixture? e.g., why do we need to manipulate the cache, why do we need to create an instance of S3FileSystem?

[martindurant]

pytest-asyncio creates a new event loop for each async test. When an async-mode s3fs instance is made from async, it will be assigned to the loop from which it is made. That means that if you use s3fs again from a subsequent test, you will have the same identical instance, but be running on a different loop - which fails.

For the rest: it's very convenient to clean up the state of the store between tests, make sure we start off blank each time.

[d-v-b]

looks good, thank you @martindurant et al