Fix transform graph - #1077

Merged
melonora merged 7 commits into
scverse:mainfrom
melonora:fix_transform_graph
Feb 21, 2026
Merged

Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom
melonora:fix_transform_graph

Conversation

@melonora

@melonoramelonora commented Feb 20, 2026

Copy link
Copy Markdown
Collaborator

Closes#1064
Closesscverse/napari-spatialdata#388

This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.
The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.

@codecov

codecovBot commented Feb 20, 2026•
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

LucaMarconato
LucaMarconato reviewed Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py Outdated
# to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
# by setting the optimization to False when performing this operation.
arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
config.set({"optimization.tune.active": True})

@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!

@melonoramelonoraFeb 20, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.

@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed the fix_transform_graph branch 2 times, most recently from bbd1efc to e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato

LucaMarconato commented Feb 21, 2026

Copy link
Copy Markdown
Member

Looks great! Two small comments:

  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?

@melonora
melonora merged commit 1383834 into scverse:mainFeb 21, 2026
9 checks passed
@LucaMarconatoLucaMarconato mentioned this pull request Mar 2, 2026
Closed
ArneDefauw pushed a commit to ArneDefauw/spatialdata that referenced this pull request Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz mentioned this pull request Mar 13, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees

No one assigned

Labels

release-fixed

Projects

None yet

Milestone

No milestone

2 participants

@melonora@LucaMarconato
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Fix transform graph - #1077
Merged
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Feb 21, 2026
Merged
Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Conversation
@melonora
@melonoramelonora
commented
Feb 20, 2026

Copy link


Copy Markdown

Collaborator
Closes#1064
Closesscverse/napari-spatialdata#388
This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.

The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.
@codecov
codecovBot
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
LucaMarconato
LucaMarconato
reviewed
Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py
Outdated

 # to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
 # by setting the optimization to False when performing this operation.
 arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
 config.set({"optimization.tune.active": True})
@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!
@melonoramelonoraFeb 20, 2026

Copy link


Copy Markdown

CollaboratorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.
@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed
the
fix_transform_graph
branch
2 times, most recently
from
bbd1efc to
e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato
LucaMarconato
commented
Feb 21, 2026

Copy link


Copy Markdown

Member
Looks great! Two small comments:
  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?
@melonora
melonora
merged commit 1383834
into
scverse:mainFeb 21, 2026

9 checks passed

@LucaMarconatoLucaMarconato
mentioned this pull request
Mar 2, 2026

Closed

ArneDefauw pushed a commit
to ArneDefauw/spatialdata
that referenced
this pull request
Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz
mentioned this pull request
Mar 13, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees


No one assigned


Labels

release-fixed

Projects


None yet


Milestone


No milestone


2 participants

@melonora@LucaMarconato

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix transform graph - #1077
Merged
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Feb 21, 2026
Merged
Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Conversation
@melonora
@melonoramelonora
commented
Feb 20, 2026

Copy link


Copy Markdown

Collaborator
Closes#1064
Closesscverse/napari-spatialdata#388
This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.

The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.
@codecov
codecovBot
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
LucaMarconato
LucaMarconato
reviewed
Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py
Outdated

 # to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
 # by setting the optimization to False when performing this operation.
 arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
 config.set({"optimization.tune.active": True})
@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!
@melonoramelonoraFeb 20, 2026

Copy link


Copy Markdown

CollaboratorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.
@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed
the
fix_transform_graph
branch
2 times, most recently
from
bbd1efc to
e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato
LucaMarconato
commented
Feb 21, 2026

Copy link


Copy Markdown

Member
Looks great! Two small comments:
  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?
@melonora
melonora
merged commit 1383834
into
scverse:mainFeb 21, 2026

9 checks passed

@LucaMarconatoLucaMarconato
mentioned this pull request
Mar 2, 2026

Closed

ArneDefauw pushed a commit
to ArneDefauw/spatialdata
that referenced
this pull request
Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz
mentioned this pull request
Mar 13, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees


No one assigned


Labels

release-fixed

Projects


None yet


Milestone


No milestone


2 participants

@melonora@LucaMarconato

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix transform graph - #1077
Merged
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Feb 21, 2026
Merged
Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Conversation
@melonora
@melonoramelonora
commented
Feb 20, 2026

Copy link


Copy Markdown

Collaborator
Closes#1064
Closesscverse/napari-spatialdata#388
This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.

The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.
@codecov
codecovBot
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
LucaMarconato
LucaMarconato
reviewed
Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py
Outdated

 # to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
 # by setting the optimization to False when performing this operation.
 arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
 config.set({"optimization.tune.active": True})
@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!
@melonoramelonoraFeb 20, 2026

Copy link


Copy Markdown

CollaboratorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.
@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed
the
fix_transform_graph
branch
2 times, most recently
from
bbd1efc to
e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato
LucaMarconato
commented
Feb 21, 2026

Copy link


Copy Markdown

Member
Looks great! Two small comments:
  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?
@melonora
melonora
merged commit 1383834
into
scverse:mainFeb 21, 2026

9 checks passed

@LucaMarconatoLucaMarconato
mentioned this pull request
Mar 2, 2026

Closed

ArneDefauw pushed a commit
to ArneDefauw/spatialdata
that referenced
this pull request
Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz
mentioned this pull request
Mar 13, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees


No one assigned


Labels

release-fixed

Projects


None yet


Milestone


No milestone


2 participants

@melonora@LucaMarconato

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Fix transform graph - #1077
Merged
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Feb 21, 2026
Merged
Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Conversation
@melonora
@melonoramelonora
commented
Feb 20, 2026

Copy link


Copy Markdown

Collaborator
Closes#1064
Closesscverse/napari-spatialdata#388
This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.

The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.
@codecov
codecovBot
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
LucaMarconato
LucaMarconato
reviewed
Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py
Outdated

 # to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
 # by setting the optimization to False when performing this operation.
 arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
 config.set({"optimization.tune.active": True})
@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!
@melonoramelonoraFeb 20, 2026

Copy link


Copy Markdown

CollaboratorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.
@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed
the
fix_transform_graph
branch
2 times, most recently
from
bbd1efc to
e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato
LucaMarconato
commented
Feb 21, 2026

Copy link


Copy Markdown

Member
Looks great! Two small comments:
  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?
@melonora
melonora
merged commit 1383834
into
scverse:mainFeb 21, 2026

9 checks passed

@LucaMarconatoLucaMarconato
mentioned this pull request
Mar 2, 2026

Closed

ArneDefauw pushed a commit
to ArneDefauw/spatialdata
that referenced
this pull request
Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz
mentioned this pull request
Mar 13, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees


No one assigned


Labels

release-fixed

Projects


None yet


Milestone


No milestone


2 participants

@melonora@LucaMarconato

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix transform graph - #1077
Merged
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Feb 21, 2026
Merged
Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Conversation
@melonora
@melonoramelonora
commented
Feb 20, 2026

Copy link


Copy Markdown

Collaborator
Closes#1064
Closesscverse/napari-spatialdata#388
This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.

The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.
@codecov
codecovBot
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
LucaMarconato
LucaMarconato
reviewed
Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py
Outdated

 # to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
 # by setting the optimization to False when performing this operation.
 arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
 config.set({"optimization.tune.active": True})
@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!
@melonoramelonoraFeb 20, 2026

Copy link


Copy Markdown

CollaboratorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.
@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed
the
fix_transform_graph
branch
2 times, most recently
from
bbd1efc to
e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato
LucaMarconato
commented
Feb 21, 2026

Copy link


Copy Markdown

Member
Looks great! Two small comments:
  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?
@melonora
melonora
merged commit 1383834
into
scverse:mainFeb 21, 2026

9 checks passed

@LucaMarconatoLucaMarconato
mentioned this pull request
Mar 2, 2026

Closed

ArneDefauw pushed a commit
to ArneDefauw/spatialdata
that referenced
this pull request
Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz
mentioned this pull request
Mar 13, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees


No one assigned


Labels

release-fixed

Projects


None yet


Milestone


No milestone


2 participants

@melonora@LucaMarconato

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix transform graph - #1077
Merged
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Feb 21, 2026
Merged
Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Conversation
@melonora
@melonoramelonora
commented
Feb 20, 2026

Copy link


Copy Markdown

Collaborator
Closes#1064
Closesscverse/napari-spatialdata#388
This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.

The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.
@codecov
codecovBot
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
LucaMarconato
LucaMarconato
reviewed
Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py
Outdated

 # to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
 # by setting the optimization to False when performing this operation.
 arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
 config.set({"optimization.tune.active": True})
@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!
@melonoramelonoraFeb 20, 2026

Copy link


Copy Markdown

CollaboratorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.
@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed
the
fix_transform_graph
branch
2 times, most recently
from
bbd1efc to
e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato
LucaMarconato
commented
Feb 21, 2026

Copy link


Copy Markdown

Member
Looks great! Two small comments:
  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?
@melonora
melonora
merged commit 1383834
into
scverse:mainFeb 21, 2026

9 checks passed

@LucaMarconatoLucaMarconato
mentioned this pull request
Mar 2, 2026

Closed

ArneDefauw pushed a commit
to ArneDefauw/spatialdata
that referenced
this pull request
Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz
mentioned this pull request
Mar 13, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees


No one assigned


Labels

release-fixed

Projects


None yet


Milestone


No milestone


2 participants

@melonora@LucaMarconato

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Fix transform graph - #1077
Merged
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Feb 21, 2026
Merged
Fix transform graph#1077
melonora merged 7 commits into
scverse:mainfrom 
melonora:fix_transform_graph
Conversation
@melonora
@melonoramelonora
commented
Feb 20, 2026

Copy link


Copy Markdown

Collaborator
Closes#1064
Closesscverse/napari-spatialdata#388
This PR fixes the problem of transforming points data where the dask dataframe has multiple partitions. This particular problem occured when writing a multipartition dataframe to parquet, reading it back in with SpatialData and then transforming the data.

The problem occured due to a partition collaps when dask expr (dask >2025.0.0) optimizes the dask graph. This specific problem is prevented in transforms by explicitly passing the current partition lengths. However, this leads to a missing dependency problem in the dask graph that is prevented by setting the dask optimization to False when performing specific operations.
@codecov
codecovBot
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 91.95%. Comparing base (bc5b2ca) to head (abf3a40).
⚠️ Report is 29 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #1077 +/- ##
==========================================
+ Coverage 91.94% 91.95% +0.01% 
==========================================
Files 51 51 Lines 7673 7685 +12 ==========================================
+ Hits 7055 7067 +12 
Misses 618 618 
Files with missing linesCoverage Δ
src/spatialdata/__init__.py100.00% <100.00%> (ø)
src/spatialdata/_core/operations/transform.py91.48% <100.00%> (+0.11%)⬆️
src/spatialdata/_utils.py85.43% <100.00%> (+0.92%)⬆️
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
LucaMarconato
LucaMarconato
reviewed
Feb 20, 2026
View reviewed changes
Comment threadsrc/spatialdata/_core/operations/transform.py
Outdated

 # to collaps of the partitions. However this causes a missing dependency problem, which for now is prevented
 # by setting the optimization to False when performing this operation.
 arrays.append(data[ax].to_dask_array(lengths=[len(part) for part in data.partitions]).reshape(-1, 1))
 config.set({"optimization.tune.active": True})
@LucaMarconatoLucaMarconatoFeb 20, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be restored to the old value, maybe it was already globally turned off. Anyway, good fix!
@melonoramelonoraFeb 20, 2026

Copy link


Copy Markdown

CollaboratorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, not that I would expect many to know how to tune this, but will change it.
@melonora
docs: add note explaining the workaround
c980ecb
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
@melonora
melonoraforce-pushed
the
fix_transform_graph
branch
2 times, most recently
from
bbd1efc to
e978fa1CompareFebruary 20, 2026 21:09
@melonora
chore: push lower bound dask version to 2025.11.0
66384c2
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
@LucaMarconato
LucaMarconato
commented
Feb 21, 2026

Copy link


Copy Markdown

Member
Looks great! Two small comments:
  • in the docs could you please add a link to the dask issue where this is discussed, since this will have to be resolved upstream as it is related to dask-expr, not spatialdata.
  • could you double check that after reading, operations like .compute() work without problem? I.e. the problem is only when reading or any downstream operation?
@melonora
melonora
merged commit 1383834
into
scverse:mainFeb 21, 2026

9 checks passed

@LucaMarconatoLucaMarconato
mentioned this pull request
Mar 2, 2026

Closed

ArneDefauw pushed a commit
to ArneDefauw/spatialdata
that referenced
this pull request
Mar 12, 2026
@melonora@ArneDefauw
Fix transform graph (scverse#1077)
153be35
Summary: fix the issue of partitions of a dask dataframe collapsing due to dask graph optimization introduced with dask expressions. This fix relies on a fix in dask introduced in version 2025.12.0, thus lower bound dependency for dask is increased. * test: transforming points with multiple partitions
* fix: transform points data with multiple partitions.
* test: add test for dask_tune contextmanager
* docs: add note explaining the workaround
This is a note explaining the workaround in case people run into the partition collaps problem due to dask graph optimization.
* chore: push lower bound dask version to 2025.11.0
Reason for this is that this dask v2025.2.0 does not allow for disabling graph optimization, but neither keeps partition size consistent. Turning optimization off was introduced in dask 2025.12.0.
* docs: add dask issue
* test: test compute can be performed
@enric-bazzenric-bazz
mentioned this pull request
Mar 13, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@LucaMarconatoLucaMarconatoLucaMarconato left review comments

Assignees


No one assigned


Labels

release-fixed

Projects


None yet


Milestone


No milestone


2 participants

@melonora@LucaMarconato