ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

ome zarr chunks - #1092

Merged
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks
Mar 20, 2026
Merged

ome zarr chunks#1092
LucaMarconato merged 11 commits into
scverse:mainfrom
ArneDefauw:fix/ome_zarr_chunks

Conversation

@ArneDefauw

Copy link
Copy Markdown
Contributor

Fix for #1090.

Note that unit tests still fail for ome-zarr ==0.14.0, due to #1091

@codecov

codecovBot commented Mar 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (6a3eef7) to head (385dd2e).
⚠️ Report is 26 commits behind head on main.

Files with missing linesPatch %Lines
src/spatialdata/_io/io_raster.py85.18%8 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1092 +/- ##
==========================================
- Coverage 91.96% 91.93% -0.04% 
==========================================
Files 51 51 Lines 7729 7772 +43 ==========================================
+ Hits 7108 7145 +37 - Misses 621 627 +6 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py95.65% <100.00%> (-0.51%)⬇️
src/spatialdata/_io/io_raster.py92.09% <85.18%> (-1.81%)⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ArneDefauw
ArneDefauw marked this pull request as ready for review March 12, 2026 09:04
@ArneDefauw

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

@ArneDefauw

ArneDefauw commented Mar 12, 2026

Copy link
Copy Markdown
ContributorAuthor

also added fix for #1091.

On linux, unit tests are failing, I will try to reproduce it

Used S0 instead of s0, for pyramid scale 0, therefore unit tests were failing on linux

@LucaMarconatoLucaMarconato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I would make some changes as described.

Comment threadsrc/spatialdata/_io/io_raster.py Outdated
Comment on lines +93 to +103
def _prepare_single_scale_storage_options(
storage_options: JSONDict | list[JSONDict] | None,
) -> JSONDict | list[JSONDict] | None:
if storage_options is None:
return None
if isinstance(storage_options, dict):
prepared = dict(storage_options)
if "chunks" in prepared:
prepared["chunks"] = _normalize_explicit_chunks(prepared["chunks"])
return prepared
return [dict(options) for options in storage_options]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function behaves like _prepare_multiscale_storage_options(), without normalizing the list of storage options case. Can we remove it and just use _prepare_multiscale_storage_options() (after renaming it to _prepare_storage_options()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I unified the two functions in 20696b5. Happy to hear what you think (in case we need two, we can revert, but I think we can proceed with one function).

Comment on lines +43 to +44
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? Please either remove or document.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +51 to +52
if isinstance(value, str | bytes):
return False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this check.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed here: 2450bd4

Comment on lines +58 to +67
def _is_regular_dask_chunk_grid(chunk_grid: Sequence[Sequence[int]]) -> bool:
# Match Dask's private _check_regular_chunks() logic without depending on its internal API.
for axis_chunks in chunk_grid:
if len(axis_chunks) <= 1:
continue
if len(set(axis_chunks[:-1])) > 1:
return False
if axis_chunks[-1] > axis_chunks[0]:
return False
return True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a docstring with examples (or examples in-line with the code) to show what fails and what not.

I would add the following:
triggers the continue in the first if:

  • [(4,)]
  • [()]

triggers the first return False

  • [(4, 4, 3, 4)]

triggers the second return False

  • [(4, 4, 4, 5)]

exits with the last return True

  • [(4, 4, 4, 4)], succeeds, all chunks equal
  • [(4, 4, 4, 1)], succeeds, final chunk is < of the initial one

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: I would add all the examples above in a test, for the function _is_regular_dask_chunk_grid().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added docstring and tests here: 2450bd4

Comment threadtests/io/test_readwrite.py Outdated
Comment on lines +629 to +634
def test_write_irregular_dask_chunks_without_explicit_storage_options(tmp_path: Path) -> None:
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})

sdata.write(tmp_path / "data.zarr")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would find it more natural if this test was failing. Now that chunks = raster_data.chunks has been removed it, writing doesn't fail, but it ignores the chunks in the data. I think a natural behavior is that if storage option specifies chunks, these are used, otherwise the ones from the data (and if the `chunks from the data are irregular and no storage options are specified, an error would be raised).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now implemented this in 20696b5 changing the test so that it expects to fail.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

I'll go ahead and implement a fix for the code review points.

@LucaMarconato

LucaMarconato commented Mar 19, 2026

Copy link
Copy Markdown
Member

@LucaMarconato

Copy link
Copy Markdown
Member

I implemented the changes mentioned in the code review. Please let me know if you agree with the changes. If yes I'll merge and work on a release.

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Hi @LucaMarconato , thanks for having a look!

My idea was to stop auto-filling storage_options["chunks"] with raster_data.data, i.e. here https://github.com/ArneDefauw/spatialdata/blob/2450bd437914004cc940279482f2dc60293e0575/src/spatialdata/_io/io_raster.py#L375

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

I still gaurded for irregular chunks, because spatialdata._io.io_raster import write_image exposes storage_options as a parameter.

I think there are valid arguments for both implementations.

If we go for auto-filling storage_options (your changes), then I would change the ValueError raised when doing e.g.:


import tempfile
from pathlib import Path
import dask.array as da
from numpy.random import default_rng
from spatialdata import SpatialData
from spatialdata.models import Image2DModel
tmpdir = Path(tempfile.mkdtemp())
RNG = default_rng(0)
data = da.from_array(RNG.random((3, 800, 1000)), chunks=((3,), (300, 200, 300), (512, 488)))
image = Image2DModel.parse(data, dims=("c", "y", "x"))
sdata = SpatialData(images={"image": image})
sdata.write(tmpdir / "data.zarr", overwrite=True)

from

ValueError: storage_options['chunks'] must be a Zarr chunk shape or a regular Dask chunk grid. Irregular Dask chunk grids must be rechunked before writing or omitted.

To specify how user can avoid it, e.g. change it to:

ValueError: storage_options["chunks"] must resolve to a Zarr chunk shape or a regular Dask chunk grid. The current raster has irregular Dask chunks, which cannot be written to Zarr. To fix this, rechunk before writing, for example by passing regular chunks=... to Image2DModel.parse(...) / Labels2DModel.parse(...).

@ArneDefauw

ArneDefauw commented Mar 20, 2026

Copy link
Copy Markdown
ContributorAuthor

Added small update to guard against storage_options["chunks"]="" and storage_options["chunks"]=b"auto" . Bit of an edge case, and the reason why this if isinstance(value, str | bytes) got introduced in

def _is_flat_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
if isinstance(value, str | bytes):
return False
if not isinstance(value, Sequence):
return False
return all(isinstance(v, int) for v in value)

Also updated the ValueErorr to be more user friendly, see previous comment

@LucaMarconato

Copy link
Copy Markdown
Member

Thanks for the comments and for the changes.

This way we let ome_zarr handling the rechunking of irregular chunks, and users of SpatialData do not need to worry about rechunking before writing to a zarr store using .write_element or .write. (ome_zarr writes a warning log message if rechunking is necessary for irregular chunks)

That's a good point! I restored the old implementation of _prepare_storage_options() that didn't depend on the data (while keeping the unified function instead of one for single-scale and one for multi-scale).

Ready to merge!

@LucaMarconato
LucaMarconato merged commit 78f75ef into scverse:mainMar 20, 2026
9 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

@ArneDefauw@LucaMarconato