Improve performance of writing pyramids

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.

[aeisenbarth]

Closes#577

• When doing multiple computations in Dask, it is better to submit them at once so that Dask can optimize the computational graph, re-use intermediate results and avoid recomputations.
• This capability was added to ome-zarr-py in #257 as a flag compute=True that can optionally be disabled to control the computation on your own. However, SpatialData kept the default enabled.
• As a consequence, every pyramid level was computed separately, and had to read again the whole input image from disk (if disk-backed). Due to this, the performance decreased with the number of scale factors:
  $$n := \textrm{number of pixels of scale level 0}$$
  $$f := \textrm{number of scale factors}$$
  $$\mathcal{O}\biggl(n \cdot (f+1)\biggr)$$

By disabling immediate computation, this PR collects all delayed objects (of pyramid levels) and lets Dask compute them at once. Input images now only need to be read once. The number of pixel read operations has now computational complexity of $$\mathcal{O}(n)$$

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.89%. Comparing base (c09f35a) to head (f167e54).
Report is 52 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #726 +/- ##
=======================================
Coverage 91.88% 91.89% =======================================
Files 45 45 Lines 6917 6919 +2 =======================================
+ Hits 6356 6358 +2 
Misses 561 561 

Files with missing lines	Coverage Δ
src/spatialdata/_io/io_raster.py	96.09% <100.00%> (+0.06%)	⬆️

[LucaMarconato]

Thanks for the PR, it looks great! I really like the test comparing the number of read-write operations.