Skip to content

GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream. - #46140

Draft
gitmodimo wants to merge 21 commits into
apache:mainfrom
gitmodimo:asof_join_pause
Draft

GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream.#46140
gitmodimo wants to merge 21 commits into
apache:mainfrom
gitmodimo:asof_join_pause

Conversation

@gitmodimo

@gitmodimogitmodimo commented Apr 14, 2025

Copy link
Copy Markdown
Contributor

Rationale for this change

Current implementation of asof join does not respect backpressure from downstream nodes. PauseProducing implementation is empty. This leads to unbound memory consumption with slow sinks.

What changes are included in this PR?

With this change PauseProducing causes ProcessThread to pause when backpressure is applied. Pause logic stolen from source_node and seems to work.

Inputs do not require additional handling sice each input has its own backpressure queue.

Are these changes tested?

Tested in my application. Still deciding how to test it. Either by checking memory consumption or by checking whether the pause is propagating upstream? The latter seems easier.
Yes

Are there any user-facing changes?

No

PS
Also I did not modify the single threaded version. Can someone using it chime in?

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@gitmodimogitmodimo changed the title [C++][Acero] Asofjoin respect PauseProducing from downstream.GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream.May 13, 2025
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #46421has been automatically assigned in GitHub to PR creator.

@gitmodimo

Copy link
Copy Markdown
ContributorAuthor

I merged @mroz45 tests for this use case.

@gitmodimo
gitmodimo marked this pull request as draft May 27, 2025 10:26
@gitmodimo
gitmodimo marked this pull request as ready for review July 4, 2025 09:27
@github-actions

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

@github-actionsgithub-actionsBot added the Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise label Aug 26, 2026
@github-actionsgithub-actionsBot removed the Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise label Aug 27, 2026
@gitmodimo
gitmodimo marked this pull request as draft August 29, 2026 07:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gitmodimo@mroz45