Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Use unsigned bytes to back Buffer - #2738

Merged
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer#2738
d-v-b merged 6 commits into
zarr-developers:mainfrom
QuLogic:unsigned-buffer

Conversation

@QuLogic

@QuLogicQuLogic commented Jan 21, 2025

Copy link
Copy Markdown
Contributor

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes#2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic

QuLogic commented Jan 21, 2025

Copy link
Copy Markdown
ContributorAuthor

PS, going to ignore tests for now.

@QuLogicQuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
@github-actionsgithub-actionsBot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby

dstansby commented Jan 28, 2025

Copy link
Copy Markdown
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjonesmaxrjones mentioned this pull request Apr 7, 2025
Closed
@maxrjones

maxrjones commented Apr 11, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., https://github.com/maxrjones/virtual-tiff/pull/4).

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

(clicking update branch in case that fixes it)

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkhamjakirkham reopened this Apr 11, 2025
@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Ok better RTD checked out the code 😌

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkhamjakirkham reopened this Apr 11, 2025
@maxrjones

maxrjones commented Apr 11, 2025

Copy link
Copy Markdown
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham

jakirkham commented Apr 11, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Merged
Open
@maxrjones

maxrjones commented Apr 18, 2025

Copy link
Copy Markdown
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b

d-v-b commented Apr 18, 2025

Copy link
Copy Markdown
Contributor

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes

@d-v-bd-v-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b
d-v-b merged commit bb55f0c into zarr-developers:mainApr 19, 2025
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic@d-v-b
@dstansby@jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.
Fixeszarr-developers#2735
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: David Stansby <dstansby@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view followingzarr-developers#2738
bf3c713
@TomAugspurgerTomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-botBot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.
Authors:
- Tom Augspurger (https://github.com/TomAugspurger)
Approvers:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Benjamin Zaitlen (https://github.com/quasiben)
URL: #699
@ziw-liuziw-liu mentioned this pull request May 5, 2025
Closed
d-v-b added a commit that referenced this pull request May 14, 2025
@tomwhite@jakirkham@d-v-b
Avoid memory copy in obstore write (#2972)
d615783
* Avoid memory copy in obstore write
* Add as_bytes_like method to Buffer
* Add changelog entry
* No need to take unsigned bytes view following #2738
* Change method name to `as_buffer_like`
---------
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@QuLogic
QuLogic deleted the unsigned-buffer branch August 23, 2025 09:41
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-bd-v-bd-v-b approved these changes

Assignees

No one assigned

Labels

needs release notesAutomatically applied to PRs which haven't added release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors

5 participants

@QuLogic@dstansby@maxrjones@jakirkham@d-v-b