Skip to content

chore: mark partial reduce stream as can handle back pressure (spillable) - #25075

Merged
rluvaton merged 1 commit into
apache:mainfrom
rluvaton:unmark-partial-hash-stream-as-spillable
Sep 9, 2026
Merged

chore: mark partial reduce stream as can handle back pressure (spillable)#25075
rluvaton merged 1 commit into
apache:mainfrom
rluvaton:unmark-partial-hash-stream-as-spillable

Conversation

@rluvaton

@rluvaton rluvaton commented Sep 8, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

While adding fuzz tests for aggregate streams I encounter OOMs that partial reduce takes up all the memory and leave nothing to partial

Same as:

// We interpret 'can spill' as 'can handle memory back pressure'.
// This value needs to be set to true for the default memory pool implementations
// to ensure fair application of back pressure amongst the memory consumers.

What changes are included in this PR?

mark reduce hash aggregate stream as spillable

What is the testing strategy for this PR?

This was found as part of:

Are there any user-facing changes?

No

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Sep 8, 2026
@rluvaton
rluvaton marked this pull request as ready for review September 8, 2026 14:23
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.74%. Comparing base (a5c809f) to head (e3b7610).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25075      +/-   ##
==========================================
- Coverage   81.74%   81.74%   -0.01%     
==========================================
  Files        1128     1128              
  Lines      416644   416645       +1     
  Branches   416644   416645       +1     
==========================================
- Hits       340592   340578      -14     
- Misses      55995    56005      +10     
- Partials    20057    20062       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@2010YOUY01 2010YOUY01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@rluvaton
rluvaton added this pull request to the merge queue Sep 9, 2026
Merged via the queue into apache:main with commit 35c56b0 Sep 9, 2026
41 checks passed
@rluvaton
rluvaton deleted the unmark-partial-hash-stream-as-spillable branch September 9, 2026 04:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants