Skip to content

feat: eliminate LEFT/RIGHT JOINs with redundant sides - #23566

Merged
neilconway merged 8 commits into
apache:mainfrom
simonvandel:push-trmumwlllzop
Aug 4, 2026
Merged

feat: eliminate LEFT/RIGHT JOINs with redundant sides#23566
neilconway merged 8 commits into
apache:mainfrom
simonvandel:push-trmumwlllzop

Conversation

@simonvandel

@simonvandelsimonvandel commented Jul 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • Closes #.

Rationale for this change

Join elimination is useful in e.g. generated queries or views, where joined columns can end up not being used.

An explanation of when the optimization is valid, can be seen in the docs update for EliminateJoin.

What changes are included in this PR?

Builds upon the analysis that #22652 introduced for the EliminateJoin optimization pass. We use it to detect when the right-side of a left-join can be removed. Same symmetrical rule for right-join elimination.

Are these changes tested?

Yes, SLT additions that test the feature end-to-end.

Are there any user-facing changes?

Yes, faster queries!

claude added 2 commits July 14, 2026 14:12
@claude@simonvandel
left-join elimination test-cases
211b676
@claude@simonvandel
feat: eliminate LEFT JOINs whose right side is unused
af7bc0e
Extend the `EliminateJoin` rule to remove a LEFT JOIN entirely
(replacing it with its left input) when:
1. none of the right side's columns are referenced above the join, and
2. the join cannot multiply left rows: either the right side is provably
unique on the equi-join keys (PRIMARY KEY / UNIQUE constraints,
GROUP BY, DISTINCT), or the join's ancestors are
duplicate-insensitive.
A LEFT JOIN preserves every left row whether or not it matches, so under
these conditions the join has no observable effect. A join filter does
not block the rewrite: for a left join it only decides whether a row is
matched or null-padded, and either way the row is emitted. Such joins
commonly appear in generated SQL and in queries over views that join in
lookup tables the query does not read.
This reuses the live-column tracking, duplicate-insensitivity context,
and join-key uniqueness analysis that `EliminateJoin` already uses to
rewrite inner joins to semi joins.
Includes sqllogictest coverage for the eliminated and preserved cases,
including result correctness for unmatched and duplicated rows.
@github-actionsgithub-actionsBot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Jul 14, 2026
neilconway
neilconway reviewed Jul 16, 2026
View reviewed changes

@neilconwayneilconway left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this -- lgtm overall!

Nit: The PR description will be the git commit message when this is squash-merged, so it's confusing to talk about "first commit" vs "second commit" there.

The existing analysis has both unit tests and SLTs; can you take a look at whether any of the existing unit tests should be extended to cover outer joins?

Comment threaddatafusion/optimizer/src/eliminate_join.rs Outdated
Comment threaddatafusion/optimizer/src/eliminate_join.rs Outdated
Comment threaddatafusion/sqllogictest/test_files/joins.slt Outdated
@simonvandelsimonvandel changed the title feat: eliminate LEFT JOINs whose right side is unusedfeat: eliminate LEFT/RIGHT JOINs with redundant sidesJul 24, 2026
@simonvandel

simonvandel commented Jul 24, 2026

Copy link
Copy Markdown
ContributorAuthor

Thank you for your review @neilconway. I have responded to your comments, and adjusted the PR description to not mention individual commits.

The existing analysis has both unit tests and SLTs; can you take a look at whether any of the existing unit tests should be extended to cover outer joins?

It's true that eliminate_join.rs has no unit tests added in this PR, so in that sense there is a gap. But I have added SLT tests for the optimization which show positive/negative cases. Do you still want unit tests as well, even if that means we now test the code redundantly?

@codecov-commenter

codecov-commenter commented Jul 24, 2026•
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.55556% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.87%. Comparing base (62650ef) to head (d2b56bc).
⚠️ Report is 1 commits behind head on main.

Files with missing linesPatch %Lines
datafusion/optimizer/src/eliminate_join.rs95.55%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #23566 +/- ##
==========================================
- Coverage 80.87% 80.87% -0.01% 
==========================================
Files 1101 1101 Lines 375765 375790 +25 Branches 375765 375790 +25 ==========================================
+ Hits 303915 303934 +19 - Misses 53747 53750 +3 - Partials 18103 18106 +3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@neilconway

neilconway commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@simonvandel Sorry for the delay in responding here!

It's true that eliminate_join.rs has no unit tests added in this PR, so in that sense there is a gap. But I have added SLT tests for the optimization which show positive/negative cases. Do you still want unit tests as well, even if that means we now test the code redundantly?

It's probably fine to skip, although the existing EliminateJoin codepaths have both unit and SLT tests, so there is precedent. I don't feel super strongly about it though.

If you can you fix the merge conflict and make a call on if you'd like to add unit tests, and I think we're good to land this.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation sql SQL Planner development-process Related to development process of DataFusion logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate substrait Changes to the substrait crate catalog Related to the catalog crate common Related to common crate execution Related to the execution crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate spark labels Aug 3, 2026
@github-actionsgithub-actionsBot removed documentation Improvements or additions to documentation sql SQL Planner development-process Related to development process of DataFusion logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate substrait Changes to the substrait crate catalog Related to the catalog crate common Related to common crate execution Related to the execution crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate spark labels Aug 3, 2026
@simonvandel

simonvandel commented Aug 3, 2026

Copy link
Copy Markdown
ContributorAuthor

@neilconway I have resolved the conflicts and decided to keep the tests as-is (coverage in SLT)

@neilconway
neilconway enabled auto-merge August 3, 2026 17:18
@neilconway
neilconway added this pull request to the merge queueAug 3, 2026
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Aug 3, 2026
@neilconway
neilconway added this pull request to the merge queueAug 4, 2026
Merged via the queue into apache:main with commit b2ac10fAug 4, 2026
40 checks passed
kosiew pushed a commit to kosiew/datafusion that referenced this pull request Aug 12, 2026
@simonvandel@claude@kosiew
feat: eliminate LEFT/RIGHT JOINs with redundant sides (apache#23566)
af05b8f
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closesapache#123` indicates that this PR will close issue apache#123.
-->
- Closes #.
## Rationale for this change
Join elimination is useful in e.g. generated queries or views, where
joined columns can end up not being used.
An explanation of when the optimization is valid, can be seen in the
docs update for `EliminateJoin`.
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Builds upon the analysis that
apache#22652 introduced for the
`EliminateJoin` optimization pass. We use it to detect when the
right-side of a left-join can be removed. Same symmetrical rule for
right-join elimination.
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
Yes, SLT additions that test the feature end-to-end.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
Yes, faster queries!
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neilconwayneilconwayneilconway approved these changes

Assignees

No one assigned

Labels

optimizerOptimizer rulessqllogictestSQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@simonvandel@codecov-commenter@neilconway@claude