Establish the high level API for sort pushdown and the optimizer rule and support reverse files and row groups by zhuqi-lucas · Pull Request #19064 · apache/datafusion

zhuqi-lucas · 2025-12-03T08:13:52Z

Which issue does this PR close?

Establish the high level API for sort pushdown and the optimizer rule. Only re-arrange files and row groups and return Inexact, now support reverse order case, and we don't need to cache anything for this implementation, so it's no memory overhead.

It will have huge performance improvement with dynamic topk pushdown to skip row groups.

Details:

Performance results on ClickBench sorted data: 13ms vs 300ms baseline (23x faster), close to aggressive caching approach (9.8ms) but with much better memory stability details

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

…. Only re-arrange files and row groups and return Inexact.

Copilot

Pull request overview

This PR establishes the high-level API for sort pushdown optimization in DataFusion, specifically targeting Parquet files. The optimization improves TopK query performance by reordering files and row groups based on statistics, though it returns inexact ordering (keeping the Sort operator for correctness while enabling early termination).

Key Changes:

Added enable_sort_pushdown configuration option (default: true) to control the optimization
Implemented the optimizer rule PushdownSort that detects SortExec nodes and pushes sort requirements down to data sources
Extended ExecutionPlan, DataSource, and FileSource traits with try_pushdown_sort() methods to support the optimization
Added reverse_scan_inexact flag to ParquetSource to reverse row group read order when beneficial

Reviewed changes

Copilot reviewed 25 out of 30 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`docs/source/user-guide/configs.md`	Added documentation for the new `enable_sort_pushdown` configuration option
`datafusion/sqllogictest/test_files/topk.slt`	Updated test expectation to show `reverse_scan_inexact=true` in explain output
`datafusion/sqllogictest/test_files/spark/bitwise/bit_count.slt`	Fixed line number formatting in test file
`datafusion/sqllogictest/test_files/slt_features.slt`	Fixed line number formatting in test file
`datafusion/sqllogictest/test_files/information_schema.slt`	Added `enable_sort_pushdown` to configuration schema output
`datafusion/sqllogictest/test_files/explain.slt`	Updated test expectations to show PushdownSort optimizer phase
`datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt`	Added comprehensive integration tests for sort pushdown with various scenarios
`datafusion/sqllogictest/test_files/create_external_table.slt`	Updated test expectation to show `reverse_scan_inexact=true`
`datafusion/proto/src/logical_plan/file_formats.rs`	Added `enable_sort_pushdown` field to protobuf serialization (with hardcoded value bug)
`datafusion/proto/src/generated/datafusion_proto_common.rs`	Added `enable_sort_pushdown` field to generated protobuf structures
`datafusion/proto-common/src/to_proto/mod.rs`	Implemented protobuf serialization for `enable_sort_pushdown` (with hardcoded value bug)
`datafusion/proto-common/src/generated/prost.rs`	Added `enable_sort_pushdown` field to generated protobuf structures
`datafusion/proto-common/src/generated/pbjson.rs`	Added JSON serialization support for `enable_sort_pushdown`
`datafusion/proto-common/src/from_proto/mod.rs`	Implemented protobuf deserialization for `enable_sort_pushdown` (with hardcoded value bug)
`datafusion/proto-common/proto/datafusion_common.proto`	Added `enable_sort_pushdown` field to protobuf schema
`datafusion/physical-plan/src/execution_plan.rs`	Added `try_pushdown_sort()` trait method to ExecutionPlan
`datafusion/physical-optimizer/src/pushdown_sort.rs`	Implemented the core PushdownSort optimizer rule
`datafusion/physical-optimizer/src/optimizer.rs`	Registered PushdownSort optimizer in the optimization pipeline
`datafusion/physical-optimizer/src/lib.rs`	Exported the new pushdown_sort module
`datafusion/execution/src/config.rs`	Added session config method (with naming and documentation issues)
`datafusion/datasource/src/source.rs`	Added `try_pushdown_sort()` trait method to DataSource and DataSourceExec
`datafusion/datasource/src/file_scan_config.rs`	Implemented sort pushdown logic for FileScanConfig including file reversal
`datafusion/datasource/src/file.rs`	Added `SortOrderPushdownResult` enum and `try_pushdown_sort()` trait method to FileSource
`datafusion/datasource-parquet/src/source.rs`	Implemented sort pushdown for ParquetSource with `reverse_scan_inexact` flag
`datafusion/datasource-parquet/src/opener.rs`	Implemented row group reversal logic in ParquetOpener
`datafusion/core/tests/physical_optimizer/test_utils.rs`	Added OptimizationTest harness for testing physical optimizers
`datafusion/core/tests/physical_optimizer/pushdown_sort.rs`	Added comprehensive unit tests for the PushdownSort optimizer
`datafusion/core/tests/physical_optimizer/mod.rs`	Registered the pushdown_sort test module
`datafusion/common/src/file_options/parquet_writer.rs`	Added `enable_sort_pushdown` field to ParquetOptions destructuring
`datafusion/common/src/config.rs`	Added `enable_sort_pushdown` configuration field (with indentation issue)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

zhuqi-lucas · 2025-12-15T13:39:09Z

I addressed all comments now, cc @alamb @adriangb.

And the latest run benchmark sql_planner shows no regression.

adriangb · 2025-12-15T16:23:34Z

+ order: &[PhysicalSortExpr],
+ ) -> Result<SortOrderPushdownResult<Arc<dyn ExecutionPlan>>> {
+ // RepartitionExec only maintains input order if preserve_order is set
+ // or if there's only one partition


Is the single partition case handled?

Yes - maintains_input_order_helper returns true for single partitions, so the check passes and sort pushdown works correctly.

adriangb · 2025-12-15T16:25:25Z

+ // Default: clone self without modification
+ // ParquetSource will override this
+ not_impl_err!("with_file_ordering_info not implemented for this FileSource")


The comment doesn't match the implementation

Good catch @adriangb , fixed the comments in latest PR, thanks!

zhuqi-lucas · 2025-12-16T06:16:09Z

Thank you @adriangb for review! I will wait for @alamb to double check before merging it, thanks!

alamb · 2025-12-16T19:13:21Z

Add push down sort to the source (table provider) Add push down sort to the source (table provider) #10433

Let me know if you want me to double check anything specific. Otherwise don't feel like you have to wait for me -- I am not likely to be able to get back here for the next few days

BTW I also added this to the list of features to highlight for 52 n

Establish the high level API for sort pushdown and the optimizer rule and support reverse files and row groups #19064

zhuqi-lucas · 2025-12-17T02:08:56Z

Add push down sort to the source (table provider) Add push down sort to the source (table provider) #10433
Let me know if you want me to double check anything specific. Otherwise don't feel like you have to wait for me -- I am not likely to be able to get back here for the next few days
BTW I also added this to the list of features to highlight for 52 n
Establish the high level API for sort pushdown and the optimizer rule and support reverse files and row groups #19064

Thank you @alamb and @adriangb for the review and discussion! I've merged the PR now. If any issues come up, I'll address them promptly. Thanks again!

adriangb · 2025-12-17T02:12:32Z

Thank you for this amazing piece of work! Very excited to see where this goes next.

alamb · 2025-12-17T22:24:34Z

100% agree -- this is so cool

xudong963 · 2025-12-18T06:55:03Z

+ match sort_input.try_pushdown_sort(required_ordering)? {
+ SortOrderPushdownResult::Exact { inner } => {
+ // Data source guarantees perfect ordering - remove the Sort operator
+ Ok(Transformed::yes(inner))


If the sort has fetch, can we directly remove the sort?

Yes @xudong963 , we will implement the remove sort part in phase 2, here is the ticket:
in #19329

This PR implemented the inexact mode, which has similar performance from sorted data clickbench.

I think the sort operator will be removed in the enforce_sorting if the required ordering is matched.

Yes agreed. I think what we ultimately need to do is return an ExecutionPlan that claims to satisfy the ordering and let the existing optimizer rule remove the sort for us.

Yes, i was testing the exact reverse cases, the enforce_sorting is not enough for the exact reverse, but it's ok for current main branch.

## Which issue does this PR close? - Closes#19403 ## Rationale for this change After #19064 was completed, I found it couldn't meet our internal project requirements: 1. **Redesign try_reverse_output using EquivalenceProperties** - The previous implementation used a simple `is_reverse` method that could only handle basic reverse matching - Now leveraging `EquivalenceProperties` can handle more complex scenarios, including constant column elimination and monotonic functions 2. **Switch to ordering_satisfy method for ordering matching** - This method internally: - Normalizes orderings (removes constant columns) - Checks monotonic functions (like `date_trunc`, `CAST`, `CEIL`) - Handles prefix matching 3. **Extend sort pushdown support to more operators** - Added `try_pushdown_sort` implementation for `ProjectionExec`, `FilterExec`, `CooperativeExec` - These operators can now pass sort requests down to their children ## What changes are included in this PR? ### Core Changes: 1. **ParquetSource::try_reverse_output** (datasource-parquet/src/source.rs) - Added `eq_properties` parameter - Reverses all orderings in equivalence properties - Uses `ordering_satisfy` to check if reversed ordering satisfies the request - Removed `file_ordering` field and `with_file_ordering_info` method 2. **FileSource trait** (datasource/src/file.rs) - Updated `try_reverse_output` signature with `eq_properties` parameter - Added detailed documentation explaining parameter usage and examples 3. **FileScanConfig::try_pushdown_sort** (datasource/src/file_scan_config.rs) - Simplified logic to directly call `file_source.try_reverse_output` - No longer needs to pre-check ordering satisfaction or set file ordering info 4. **New operator support** - `FilterExec::try_pushdown_sort` - Pushes sort below filters - `ProjectionExec::try_pushdown_sort` - Pushes sort below projections - `CooperativeExec::try_pushdown_sort` - Supports sort pushdown in cooperative execution 5. **Removed obsolete methods** - Deleted `LexOrdering::is_reverse` - replaced by `ordering_satisfy` ### Feature Enhancements: **Supported optimization scenarios:** 1. **Constant column elimination** (Test 7) ```sql -- File ordering: [timeframe ASC, period_end ASC] -- Query: WHERE timeframe = 'quarterly' ORDER BY period_end DESC -- Effect: After timeframe becomes constant, reverse scan is enabled ``` 2. **Monotonic function support** (Test 8) ```sql -- File ordering: [ts ASC] -- Query: ORDER BY date_trunc('month', ts) DESC -- Effect: date_trunc is monotonic, reverse scan satisfies the request ``` ## Are these changes tested? Yes, comprehensive tests have been added: - **Test 7 (237 lines)**: Constant column elimination scenarios - Single constant column filter - Multi-value IN clauses (doesn't trigger optimization) - Literal constants in sort expressions - Non-leading column filters (edge cases) - **Test 8 (355 lines)**: Monotonic function scenarios - `date_trunc` (date truncation) - `CAST` (type conversion) - `CEIL` (ceiling) - `ABS` (negative case - not monotonic over mixed positive/negative range) All tests verify: - Presence of `reverse_row_groups=true` in physical plans - Correctness of query results ## Are there any user-facing changes? **API Changes:** - `FileSource::try_reverse_output` signature changed (added `eq_properties` parameter) - Removed `FileSource::with_file_ordering_info` method - Removed `LexOrdering::is_reverse` public method **User-visible improvements:** - More queries can leverage reverse row group scanning for optimization - Especially queries with `WHERE` clauses that make certain columns constant - Queries using monotonic functions (like date functions, type conversions) **Note:** This PR returns `Inexact` results because only row group order is reversed, not row order within row groups. Future enhancements could include: - File reordering based on statistics (returning `Exact`) - Partial sort pushdown for prefix matches --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

zhuqi-lucas added 2 commits December 3, 2025 16:13

Establish the high level API for sort pushdown and the optimizer rule…
36d2785
…. Only re-arrange files and row groups and return Inexact.

fix
85bfd74

zhuqi-lucas added 2 commits December 3, 2025 17:28

Add topk test
df4f042

fix format for tree
b9b889d

zhuqi-lucas mentioned this pull request Dec 3, 2025
Support reverse parquet scan and fast parquet order inversion at row group level #18817
Closed

zhuqi-lucas marked this pull request as ready for review December 3, 2025 09:53

CopilotAI review requested due to automatic review settings December 3, 2025 09:53

Copilotstarted reviewing on behalf of zhuqi-lucas December 3, 2025 09:53 View session

zhuqi-lucas changed the title ~~Draft Establish the high level API for sort pushdown and the optimizer rule. Only re-arrange files and row groups and return Inexact.~~Establish the high level API for sort pushdown and the optimizer rule and support reverse files and row groups Dec 3, 2025

Copilotfinished reviewing on behalf of zhuqi-lucas December 3, 2025 09:55

zhuqi-lucas requested review from 2010YOUY01, adriangb and alamb December 3, 2025 09:57

CopilotAI reviewed Dec 3, 2025
View reviewed changes

martin-g reviewed Dec 3, 2025
View reviewed changes

zhuqi-lucasand others added 6 commits December 4, 2025 11:26

Update datafusion/execution/src/config.rs
595f159
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update datafusion/proto/src/logical_plan/file_formats.rs
84e4d70
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update datafusion/proto-common/src/to_proto/mod.rs
dcfa3a6
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update datafusion/proto-common/src/from_proto/mod.rs
235dcdb
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update datafusion/common/src/config.rs
0fe1d77
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update datafusion/proto/src/logical_plan/file_formats.rs
6be3d74
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

zhuqi-lucas mentioned this pull request Dec 15, 2025
Add ParquetOpenerBuilder to make code more clean and readable #19333
Closed

zhuqi-lucasand others added 2 commits December 15, 2025 21:23

Merge branch 'main' into issue_19059
369eef1

introduce PreparedAccessPlan to make code clear
6f69094

adriangb reviewed Dec 15, 2025
View reviewed changes

adriangb approved these changes Dec 15, 2025
View reviewed changes

fix comments
1e3d07c

Merge branch 'main' into issue_19059
bb0ef19

alamb mentioned this pull request Dec 16, 2025
Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) #18566
Closed
41 tasks

Merge branch 'main' into issue_19059
43a2280

zhuqi-lucas added this pull request to the merge queue Dec 17, 2025

Merged via the queue into apache:main with commit 3a41cc6Dec 17, 2025
14 of 15 checks passed

xudong963 reviewed Dec 18, 2025
View reviewed changes

alamb mentioned this pull request Dec 18, 2025
Push down preferred sorts into TableScan logical plan node #17337
Closed

This was referenced Dec 19, 2025
Add more ExecutionPlan to support sort pushdown #19394
Open
Redesign the try_reverse_output to support more cases #19403
Closed

Nithurshen mentioned this pull request Dec 22, 2025
feat: Support sort prefix removal for logical constants in Sort Pushdown #19445
Closed
2 tasks

zhuqi-lucas mentioned this pull request Dec 22, 2025
Redesign the try_reverse_output to support more cases #19446
Merged

lonless9 mentioned this pull request Jan 26, 2026
chore: datafusion 52.1.0 lakehq/sail#1174
Merged

This was referenced Apr 6, 2026
[Epic] Dynamic row group pruning using TopK threshold during parquet scan #21399
Closed
Add benchmark for sort pushdown Inexact path (row group reorder) #21582
Closed

zhuqi-lucas mentioned this pull request May 14, 2026
Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion apache/datafusion-site#186
Merged
4 tasks

zhuqi-lucas mentioned this pull request Jun 19, 2026
[EPIC] Sort Pushdown: skip sorts and skip IO / decode for ORDER BY / TopK queries #23036
Closed

Uh oh!

Conversation

zhuqi-lucas commented Dec 3, 2025• edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhuqi-lucas commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhuqi-lucas commented Dec 16, 2025

Uh oh!

alamb commented Dec 16, 2025

Uh oh!

Uh oh!

zhuqi-lucas commented Dec 17, 2025

Uh oh!

adriangb commented Dec 17, 2025

Uh oh!

alamb commented Dec 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zhuqi-lucas commented Dec 3, 2025•
edited by alamb
Loading