Skip to content

feat: allow Full joins to reuse range co-partitioning in HashJoinExec - #23583

Merged
gabotechs merged 5 commits into
apache:mainfrom
mattp5657:feat/full-join-range-copartitioning
Jul 20, 2026
Merged

feat: allow Full joins to reuse range co-partitioning in HashJoinExec#23583
gabotechs merged 5 commits into
apache:mainfrom
mattp5657:feat/full-join-range-copartitioning

Conversation

@mattp5657

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

#23184 let compatible range-partitioned inputs satisfy inner partitioned hash joins without repartitioning. Full partitioned equi joins still always went through the conservative hash-repartition path, even when both inputs were already co-partitioned by range on the join key(s). The per-partition unmatched-row tracking in HashJoinExec is already partition-local under PartitionMode::Partitioned (not shared globally like in CollectLeft), so Full-join semantics generalize cleanly to range co-partitioning with no additional bookkeeping required.

What changes are included in this PR?

  • Extend HashJoinExec::input_distribution_requirements() to opt JoinType::Full in to allow_range_satisfaction_for_key_partitioning(), alongside the existing JoinType::Inner case. The underlying co_partitioned / compatible_co_partitioning_layout / co_partitioning_satisfied logic in distribution_requirements.rs was already join-type-agnostic, so no changes were needed there.
  • Add planner unit tests in enforce_distribution.rs covering both the compatible-layout case (no repartition inserted) and the incompatible-split-points case (repartition still inserted) for JoinType::Full.
  • Add a range_partitioned_sparse sqllogictest fixture table with the same partition layout as range_partitioned but only partially overlapping keys, and add sqllogictest coverage in range_partitioning.slt for: a compatible Full join avoiding repartition, an incompatible Full join still repartitioning, and matched/left-only/right-only unmatched rows produced correctly by a co-partitioned Full join.

Are these changes tested?

Yes:

  • Two new Rust unit tests in datafusion/core/tests/physical_optimizer/enforce_distribution.rs assert on the physical plan shape (repartition inserted or not) for compatible and incompatible range layouts.
  • Three new sqllogictest cases in datafusion/sqllogictest/test_files/range_partitioning.slt exercise the feature end-to-end against real data, including matched rows, left-only unmatched rows, and right-only unmatched rows for a Full outer join.
  • Existing enforce_distribution tests, range_partitioning.slt, and proto roundtrip tests all continue to pass.

Are there any user-facing changes?

Yes: EXPLAIN output for Full joins over compatible range-partitioned inputs will no longer show a RepartitionExec, and such queries will avoid the associated hash-shuffle cost at execution time. No public API changes.

@github-actionsgithub-actionsBot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jul 14, 2026
@mattp5657
mattp5657 marked this pull request as ready for review July 14, 2026 19:49
@mattp5657mattp5657 changed the title Feat/full join range copartitioningfeat: allow Full joins to reuse range co-partitioning in HashJoinExecJul 14, 2026

@gene-bordegaraygene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good, just rebase and ping me for a re-review before I approve 👍

@mattp5657
mattp5657force-pushed the feat/full-join-range-copartitioning branch from d2ca127 to 8e7a4daCompareJuly 15, 2026 23:35
@mattp5657

Copy link
Copy Markdown
ContributorAuthor

@gene-bordegaray This should be ready.

@gene-bordegaraygene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be good after next addresses 👍

# TEST 12: Non-Inner Range Join Repartitions
# Only inner partitioned hash joins opt in to Range satisfying KeyPartitioned
# requirements. Non-inner joins keep using Hash repartitioning.
# TEST 12: Non-Inner, Non-Full Range Join Repartitions

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets just have this say "Unsupported"

02)--RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4
03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=csv, has_header=false
04)--RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4
05)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(15), (20), (30)], 4), file_type=csv, has_header=false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add query results

@mattp5657

Copy link
Copy Markdown
ContributorAuthor

@gene-bordegaray Thank you for the comments, this should be ready when you have a chance :)

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.67%. Comparing base (4c20fa7) to head (01d6200).

Additional details and impacted files
@@ Coverage Diff @@## main #23583 +/- ##
==========================================
- Coverage 80.67% 80.67% -0.01% 
==========================================
Files 1087 1087 Lines 367541 367566 +25 Branches 367541 367566 +25 ==========================================
+ Hits 296520 296529 +9 - Misses 53365 53376 +11 - Partials 17656 17661 +5 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gene-bordegaraygene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you 🙇

@gene-bordegaray

Copy link
Copy Markdown
Contributor

cc: @gabotechs

@gabotechsgabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gabotechs
gabotechs added this pull request to the merge queueJul 20, 2026
Merged via the queue into apache:main with commit 0f5790aJul 20, 2026
40 checks passed
gene-bordegaray pushed a commit to gene-bordegaray/datafusion that referenced this pull request Jul 24, 2026
…apache#23583)
## Which issue does this PR close?
- Closesapache#23454.
## Rationale for this change
apache#23184 let compatible range-partitioned inputs satisfy inner partitioned
hash joins without repartitioning. Full partitioned equi joins still
always went through the conservative hash-repartition path, even when
both inputs were already co-partitioned by range on the join key(s). The
per-partition unmatched-row tracking in `HashJoinExec` is already
partition-local under `PartitionMode::Partitioned` (not shared globally
like in `CollectLeft`), so Full-join semantics generalize cleanly to
range co-partitioning with no additional bookkeeping required.
## What changes are included in this PR?
- Extend `HashJoinExec::input_distribution_requirements()` to opt
`JoinType::Full` in to
`allow_range_satisfaction_for_key_partitioning()`, alongside the
existing `JoinType::Inner` case. The underlying `co_partitioned` /
`compatible_co_partitioning_layout` / `co_partitioning_satisfied` logic
in `distribution_requirements.rs` was already join-type-agnostic, so no
changes were needed there.
- Add planner unit tests in `enforce_distribution.rs` covering both the
compatible-layout case (no repartition inserted) and the
incompatible-split-points case (repartition still inserted) for
`JoinType::Full`.
- Add a `range_partitioned_sparse` sqllogictest fixture table with the
same partition layout as `range_partitioned` but only partially
overlapping keys, and add sqllogictest coverage in
`range_partitioning.slt` for: a compatible Full join avoiding
repartition, an incompatible Full join still repartitioning, and
matched/left-only/right-only unmatched rows produced correctly by a
co-partitioned Full join.
## Are these changes tested?
Yes:
- Two new Rust unit tests in
`datafusion/core/tests/physical_optimizer/enforce_distribution.rs`
assert on the physical plan shape (repartition inserted or not) for
compatible and incompatible range layouts.
- Three new sqllogictest cases in
`datafusion/sqllogictest/test_files/range_partitioning.slt` exercise the
feature end-to-end against real data, including matched rows, left-only
unmatched rows, and right-only unmatched rows for a Full outer join.
- Existing `enforce_distribution` tests, `range_partitioning.slt`, and
proto roundtrip tests all continue to pass.
## Are there any user-facing changes?
Yes: `EXPLAIN` output for `Full` joins over compatible range-partitioned
inputs will no longer show a `RepartitionExec`, and such queries will
avoid the associated hash-shuffle cost at execution time. No public API
changes.
---------
Co-authored-by: Matthew Patton <matthewpatton@macbookpro.mynetworksettings.com>
gene-bordegaray pushed a commit to DataDog/datafusion that referenced this pull request Aug 7, 2026
…apache#23583)
## Which issue does this PR close?
- Closesapache#23454.
## Rationale for this change
apache#23184 let compatible range-partitioned inputs satisfy inner partitioned
hash joins without repartitioning. Full partitioned equi joins still
always went through the conservative hash-repartition path, even when
both inputs were already co-partitioned by range on the join key(s). The
per-partition unmatched-row tracking in `HashJoinExec` is already
partition-local under `PartitionMode::Partitioned` (not shared globally
like in `CollectLeft`), so Full-join semantics generalize cleanly to
range co-partitioning with no additional bookkeeping required.
## What changes are included in this PR?
- Extend `HashJoinExec::input_distribution_requirements()` to opt
`JoinType::Full` in to
`allow_range_satisfaction_for_key_partitioning()`, alongside the
existing `JoinType::Inner` case. The underlying `co_partitioned` /
`compatible_co_partitioning_layout` / `co_partitioning_satisfied` logic
in `distribution_requirements.rs` was already join-type-agnostic, so no
changes were needed there.
- Add planner unit tests in `enforce_distribution.rs` covering both the
compatible-layout case (no repartition inserted) and the
incompatible-split-points case (repartition still inserted) for
`JoinType::Full`.
- Add a `range_partitioned_sparse` sqllogictest fixture table with the
same partition layout as `range_partitioned` but only partially
overlapping keys, and add sqllogictest coverage in
`range_partitioning.slt` for: a compatible Full join avoiding
repartition, an incompatible Full join still repartitioning, and
matched/left-only/right-only unmatched rows produced correctly by a
co-partitioned Full join.
## Are these changes tested?
Yes:
- Two new Rust unit tests in
`datafusion/core/tests/physical_optimizer/enforce_distribution.rs`
assert on the physical plan shape (repartition inserted or not) for
compatible and incompatible range layouts.
- Three new sqllogictest cases in
`datafusion/sqllogictest/test_files/range_partitioning.slt` exercise the
feature end-to-end against real data, including matched rows, left-only
unmatched rows, and right-only unmatched rows for a Full outer join.
- Existing `enforce_distribution` tests, `range_partitioning.slt`, and
proto roundtrip tests all continue to pass.
## Are there any user-facing changes?
Yes: `EXPLAIN` output for `Full` joins over compatible range-partitioned
inputs will no longer show a `RepartitionExec`, and such queries will
avoid the associated hash-shuffle cost at execution time. No public API
changes.
---------
Co-authored-by: Matthew Patton <matthewpatton@macbookpro.mynetworksettings.com>
kosiew pushed a commit to kosiew/datafusion that referenced this pull request Aug 12, 2026
…apache#23583)
## Which issue does this PR close?
- Closesapache#23454.
## Rationale for this change
apache#23184 let compatible range-partitioned inputs satisfy inner partitioned
hash joins without repartitioning. Full partitioned equi joins still
always went through the conservative hash-repartition path, even when
both inputs were already co-partitioned by range on the join key(s). The
per-partition unmatched-row tracking in `HashJoinExec` is already
partition-local under `PartitionMode::Partitioned` (not shared globally
like in `CollectLeft`), so Full-join semantics generalize cleanly to
range co-partitioning with no additional bookkeeping required.
## What changes are included in this PR?
- Extend `HashJoinExec::input_distribution_requirements()` to opt
`JoinType::Full` in to
`allow_range_satisfaction_for_key_partitioning()`, alongside the
existing `JoinType::Inner` case. The underlying `co_partitioned` /
`compatible_co_partitioning_layout` / `co_partitioning_satisfied` logic
in `distribution_requirements.rs` was already join-type-agnostic, so no
changes were needed there.
- Add planner unit tests in `enforce_distribution.rs` covering both the
compatible-layout case (no repartition inserted) and the
incompatible-split-points case (repartition still inserted) for
`JoinType::Full`.
- Add a `range_partitioned_sparse` sqllogictest fixture table with the
same partition layout as `range_partitioned` but only partially
overlapping keys, and add sqllogictest coverage in
`range_partitioning.slt` for: a compatible Full join avoiding
repartition, an incompatible Full join still repartitioning, and
matched/left-only/right-only unmatched rows produced correctly by a
co-partitioned Full join.
## Are these changes tested?
Yes:
- Two new Rust unit tests in
`datafusion/core/tests/physical_optimizer/enforce_distribution.rs`
assert on the physical plan shape (repartition inserted or not) for
compatible and incompatible range layouts.
- Three new sqllogictest cases in
`datafusion/sqllogictest/test_files/range_partitioning.slt` exercise the
feature end-to-end against real data, including matched rows, left-only
unmatched rows, and right-only unmatched rows for a Full outer join.
- Existing `enforce_distribution` tests, `range_partitioning.slt`, and
proto roundtrip tests all continue to pass.
## Are there any user-facing changes?
Yes: `EXPLAIN` output for `Full` joins over compatible range-partitioned
inputs will no longer show a `RepartitionExec`, and such queries will
avoid the associated hash-shuffle cost at execution time. No public API
changes.
---------
Co-authored-by: Matthew Patton <matthewpatton@macbookpro.mynetworksettings.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratephysical-planChanges to the physical-plan cratesqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow co-partitioned Partitioning::Range inputs for full hash joins

4 participants

@mattp5657@codecov-commenter@gene-bordegaray@gabotechs