Avoid memory copy in obstore write

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.

[tomwhite]

This removes an unnecessary memory copy when writing using obstore, similar to #2944

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[jakirkham]

Thanks Tom! 🙏

Had a question below

[jakirkham]

Is it worthwhile to make this a standard method? This would make it easier to use throughout the codebase

Some possible names:

• to_bytelike
• to_binary
• to_uint8s
• ?

[d-v-b]

i guess i'm also curious to know why we have a to_bytes() method that makes a copy, when .as_numpy_array().view(np.uint8) achieves the same effective result (an iterable of bytes) without a copy.

[tomwhite]

This was suggested in #2925, so I've added an as_bytes_like method.

[jakirkham]

bytes objects always own their memory. So creating a new bytes object means making a copy

If we prefer to move away from this approach, we could add a deprecation cycle to move from to_bytes to as_bytes_like

[tomwhite]

I don't think FsspecStore works with BytesLike since it didn't accept a memoryview object when I tried it. So we probably need to keep to_bytes().

[jakirkham]

Think this would be a good issue to raise with fsspec if we can reproduce with it directly

If not, a Zarr issue would be welcome

Ideally these copies should be avoidable in the FsspecStore case as well

Edit: Ofc this is non-blocking, just would like to improve the performance in all cases

[tomwhite]

Opened fsspec/s3fs#959

[kylebarron]

Some possible names:

I'd also suggest as_buffer_like, because in my head "bytes" in Python is a type that always owns its own memory, while "buffer" doesn't necessarily own its memory, as in the buffer protocol or collections.abc.Buffer.

[d-v-b]

@tomwhite anything new here or is it ready to go in?

[tomwhite]

@d-v-b This is ready to go in I think.