Skip to content

Improve ExternalSorter ResourcesExhausted Error Message - #20226

Merged
alamb merged 2 commits into
apache:mainfrom
erenavsarogullari:improve_external_sorter_error_message_when_resource_exhausted
Feb 12, 2026
Merged

Improve ExternalSorter ResourcesExhausted Error Message#20226
alamb merged 2 commits into
apache:mainfrom
erenavsarogullari:improve_external_sorter_error_message_when_resource_exhausted

Conversation

@erenavsarogullari

@erenavsarogullarierenavsarogullari commented Feb 9, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

When there is not enough memory to continue external sort, either increasing the memory limit or decreasing sort_spill_reservation_bytes need to be applied. It can be useful to guide the user with clearer error message by highlighting required configs for the consistency because expected settings are as follows:

SET datafusion.runtime.memory_limit = '10G'
SET datafusion.execution.sort_spill_reservation_bytes = 10485760

Current:

Not enough memory to continue external sort. Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes

New:

Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.

Are these changes tested?

A new UT case has been added to cover this use-case and it has been tested locally successfully.

For updated snapshot files:
Following integration tests have been successful by DataFusion Test Pipeline:

test test_cli_top_memory_consumers::case_1 ... ok
test test_cli_top_memory_consumers::case_2 ... ok
test test_cli_top_memory_consumers::case_3 ... ok

Ref:https://github.com/apache/datafusion/actions/runs/21811797863/job/62925363536?pr=20226

Are there any user-facing changes?

Yes, improving legacy ExternalSorterResourcesExhausted Error Message.

@github-actionsgithub-actionsBot added the physical-plan Changes to the physical-plan crate label Feb 9, 2026
@erenavsarogullari
erenavsarogullariforce-pushed the improve_external_sorter_error_message_when_resource_exhausted branch 5 times, most recently from ba97e75 to 64eaa60CompareFebruary 9, 2026 03:56
@erenavsarogullari
erenavsarogullariforce-pushed the improve_external_sorter_error_message_when_resource_exhausted branch 2 times, most recently from 9539d82 to e306fe7CompareFebruary 10, 2026 17:36
kosiew
kosiew approved these changes Feb 11, 2026
View reviewed changes

@kosiewkosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erenavsarogullari

Thanks for working on this.

Left some non-blocking comments.

Comment threaddatafusion/physical-plan/src/sorts/sort.rs Outdated
Comment threaddatafusion/physical-plan/src/sorts/sort.rs Outdated
Comment threaddatafusion/physical-plan/src/sorts/sort.rs Outdated
@erenavsarogullari

erenavsarogullari commented Feb 12, 2026

Copy link
Copy Markdown
MemberAuthor

Thanks @kosiew and @martin-g for the reviews. All comments have been addressed.

@erenavsarogullari
erenavsarogullariforce-pushed the improve_external_sorter_error_message_when_resource_exhausted branch from 010cb74 to 080bf49CompareFebruary 12, 2026 05:29
@erenavsarogullari
erenavsarogullariforce-pushed the improve_external_sorter_error_message_when_resource_exhausted branch from 080bf49 to ac88bbbCompareFebruary 12, 2026 05:34
@alamb
alamb added this pull request to the merge queueFeb 12, 2026
@alamb

alamb commented Feb 12, 2026

Copy link
Copy Markdown
Contributor

🚀

Merged via the queue into apache:main with commit 4d06c40Feb 12, 2026
32 checks passed
This was referenced Feb 15, 2026
Closed
Merged
github-merge-queueBot pushed a commit that referenced this pull request Mar 3, 2026
@erenavsarogullari
refactor: Set expected runtime config in error message when the used …
476c200
…disk space during the spilling process has exceeded the allocation limit (#20375)
## Which issue does this PR close?
- Closes#20373.
## Rationale for this change
Minor refactoring on error message by exposing required config name for
the end user. This is follow-up PR to both PR: #20226 and #20372 by
runtime config: `datafusion.runtime.max_temp_directory_size`.
## What changes are included in this PR?
**Current:**
```
The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration. ```
**New:**
```
The used disk space during the spilling process has exceeded the allowable limit of {}. \
Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
```
## Are these changes tested?
Yes, legacy UT case has been updated by covering expected config name.
## Are there any user-facing changes?
Yes, error message has been updated which is exposed to end-users.
de-bgunter pushed a commit to de-bgunter/datafusion that referenced this pull request Mar 24, 2026
@erenavsarogullari@de-bgunter
Improve ExternalSorter ResourcesExhausted Error Message (apache#20226)
a0e14d2
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closesapache#123` indicates that this PR will close issue apache#123.
-->
- Closesapache#20225.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
When there is not enough memory to continue external sort, either
increasing the `memory limit` or decreasing
`sort_spill_reservation_bytes` need to be applied. It can be useful to
guide the user with clearer error message by highlighting required
configs for the consistency because expected settings are as follows:
```
SET datafusion.runtime.memory_limit = '10G'
SET datafusion.execution.sort_spill_reservation_bytes = 10485760
```
Current:
```
Not enough memory to continue external sort. Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes
```
New:
```
Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
```
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
A new UT case has been added to cover this use-case and it has been
tested locally successfully.
**For updated snapshot files:**
Following integration tests have been successful by DataFusion Test
Pipeline:
```
test test_cli_top_memory_consumers::case_1 ... ok
test test_cli_top_memory_consumers::case_2 ... ok
test test_cli_top_memory_consumers::case_3 ... ok
```
**Ref:**
https://github.com/apache/datafusion/actions/runs/21811797863/job/62925363536?pr=20226
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
Yes, improving legacy `ExternalSorter` `ResourcesExhausted` Error
Message.
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
de-bgunter pushed a commit to de-bgunter/datafusion that referenced this pull request Mar 24, 2026
@erenavsarogullari@de-bgunter
refactor: Set expected runtime config in error message when the used …
2cbb171
…disk space during the spilling process has exceeded the allocation limit (apache#20375)
## Which issue does this PR close?
- Closesapache#20373.
## Rationale for this change
Minor refactoring on error message by exposing required config name for
the end user. This is follow-up PR to both PR: apache#20226 and apache#20372 by
runtime config: `datafusion.runtime.max_temp_directory_size`.
## What changes are included in this PR?
**Current:**
```
The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration. ```
**New:**
```
The used disk space during the spilling process has exceeded the allowable limit of {}. \
Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
```
## Are these changes tested?
Yes, legacy UT case has been updated by covering expected config name.
## Are there any user-facing changes?
Yes, error message has been updated which is exposed to end-users.
@surendrachoudhary11surendrachoudhary11 mentioned this pull request Jun 18, 2026
Open
8 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@martin-gmartin-gmartin-g approved these changes
@kosiewkosiewkosiew approved these changes

Assignees

No one assigned

Labels

physical-planChanges to the physical-plan crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve ExternalSorter ResourcesExhausted Error Message

4 participants

@erenavsarogullari@alamb@martin-g@kosiew