Skip to content

[Feature](join) support LEFT_SEMI_DIRECT_RETURN_OPT - #59832

Merged
BiteTheDDDDt merged 2 commits into
apache:masterfrom
BiteTheDDDDt:dev_left_semi
Feb 6, 2026
Merged

[Feature](join) support LEFT_SEMI_DIRECT_RETURN_OPT#59832
BiteTheDDDDt merged 2 commits into
apache:masterfrom
BiteTheDDDDt:dev_left_semi

Conversation

@BiteTheDDDDt

@BiteTheDDDDtBiteTheDDDDt commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

mysql>selectcount(k1),count(distinct k1) from d_table;
+-----------+--------------------+
| count(k1) | count(distinct k1) |
+-----------+--------------------+
| 100000000 | 100 |
+-----------+--------------------+1 row inset (0.20 sec)
mysql>selectcount(k1),count(distinct k1) from dd_table;
+-----------+--------------------+
| count(k1) | count(distinct k1) |
+-----------+--------------------+
| 100 | 100 |
+-----------+--------------------+1 row inset (0.02 sec)
mysql>set enable_left_semi_direct_return_opt=false; Query OK, 0 rows affected (0.00 sec)
mysql>selectcount(*) from d_table left semi join dd_table as r ond_table.k1=r.k1;
+-----------+
| count(*) |
+-----------+
| 100000000 |
+-----------+1 row inset (0.60 sec)
mysql>set enable_left_semi_direct_return_opt=true;
Query OK, 0 rows affected (0.00 sec)
mysql>selectcount(*) from d_table left semi join dd_table as r ond_table.k1=r.k1;
+-----------+
| count(*) |
+-----------+
| 100000000 |
+-----------+1 row inset (0.38 sec)

This pull request introduces a new optimization for left semi joins with runtime filters, allowing for a "direct return" path when certain conditions are met. It also refactors runtime filter selectivity logic to improve accuracy and maintainability, and adds mechanisms to detect and propagate "IN" filters for more efficient query execution. The most important changes are grouped below:

Left Semi Join Direct Return Optimization

  • Added a left_semi_direct_return flag to HashJoinSharedState, enabling the left semi join to bypass hash table probing and directly return results when a local IN filter is detected and only a single join conjunct is present. This includes logic to detect such filters and propagate the flag through the pipeline. [1][2][3][4][5][6][7][8]

Runtime Filter Selectivity Refactoring

  • Replaced legacy selectivity tracking variables in ColumnPredicate with a new RuntimeFilterSelectivity member, delegating all selectivity logic (including resets, updates, and always_true checks) to this new class for better encapsulation and flexibility. [1][2][3][4][5]
  • Refactored RuntimeFilterSelectivity to allow dynamic configuration of sampling frequency, moved selectivity logic into instance methods, and made the always-true logic depend on the configured sampling frequency rather than a global config. [1][2][3][4]

Runtime Filter "IN" Filter Detection and Propagation

  • Implemented detection of local IN filters in RuntimeFilterProducer and RuntimeFilterProducerHelper, exposing methods to check for such filters and using this information to enable the direct return optimization. [1][2][3]
  • Added a mechanism in the runtime filter consumer to set the selectivity sampling frequency based on whether an IN filter was detected, ensuring correct selectivity handling for these cases.
  • Added state and methods in RuntimeFilterWrapper to track and expose detection of IN filters. [1][2]

Runtime Filter Wait Logic Improvement

  • Changed the wait logic in RuntimeFilterConsumer to wait indefinitely not only for bitmap filters but also when there is no remote target, ensuring correctness in more scenarios.

Minor Code Cleanups

  • Removed an unnecessary include of common/config.h from runtime_filter_selectivity.h.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 25.00% (11/44) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage52.99% (18974/35807)
Line Coverage39.04% (175841/450423)
Region Coverage33.62% (136116/404889)
Branch Coverage34.65% (58861/169857)

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (44/44) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage72.26% (25357/35092)
Line Coverage59.06% (265715/449891)
Region Coverage53.96% (220921/409424)
Branch Coverage55.56% (94803/170636)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.57% (1784/2242)
Line Coverage64.75% (31726/48997)
Region Coverage65.42% (15780/24120)
Branch Coverage56.00% (8382/14968)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.59% (1786/2244)
Line Coverage64.76% (31781/49075)
Region Coverage65.46% (15831/24185)
Branch Coverage55.99% (8406/15014)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.59% (1786/2244)
Line Coverage64.76% (31793/49093)
Region Coverage65.48% (15844/24197)
Branch Coverage56.04% (8418/15022)

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.59% (1786/2244)
Line Coverage64.78% (31802/49093)
Region Coverage65.45% (15838/24197)
Branch Coverage56.00% (8413/15022)

@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 97.65% (83/85) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.36% (25407/35604)
Line Coverage53.89% (265767/493202)
Region Coverage51.49% (220334/427884)
Branch Coverage52.96% (94722/178848)

1 similar comment
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 97.65% (83/85) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.36% (25407/35604)
Line Coverage53.89% (265767/493202)
Region Coverage51.49% (220334/427884)
Branch Coverage52.96% (94722/178848)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 97.65% (83/85) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.38% (25414/35604)
Line Coverage53.91% (265907/493202)
Region Coverage51.54% (220521/427884)
Branch Coverage52.99% (94776/178848)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 97.65% (83/85) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.39% (25417/35604)
Line Coverage53.93% (265965/493202)
Region Coverage51.54% (220529/427884)
Branch Coverage53.00% (94796/178848)

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

Comment threadbe/src/pipeline/exec/hashjoin_build_sink.h
Comment threadbe/src/runtime_filter/runtime_filter_consumer.h
Comment threadbe/src/runtime_filter/runtime_filter_consumer.cpp Outdated
Comment threadbe/src/runtime_filter/runtime_filter_selectivity.h
Comment threadbe/src/runtime_filter/runtime_filter_producer_helper.cpp Outdated
Comment threadbe/src/runtime_filter/runtime_filter_wrapper.h Outdated
update
add conf
update
fix
fix ut
fix
update
Update regression-test/suites/query_p0/join/test_left_join1.groovy
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Update be/src/runtime_filter/runtime_filter_producer.h
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Update fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Update be/src/pipeline/exec/hashjoin_probe_operator.cpp
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Update fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Update be/src/pipeline/dependency.h
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
update
@BiteTheDDDDt

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Feb 5, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 97.96% (96/98) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage73.36% (26508/36132)
Line Coverage56.44% (281173/498165)
Region Coverage54.15% (235501/434926)
Branch Coverage55.79% (101308/181604)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

Comment threadbe/src/runtime_filter/runtime_filter_consumer.h
@yiguoleiyiguolei added usercase Important user case type label dev/4.0.x labels Feb 6, 2026
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BiteTheDDDDt
BiteTheDDDDt merged commit 5275e0e into apache:masterFeb 6, 2026
31 of 33 checks passed
BiteTheDDDDt added a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Feb 6, 2026
This pull request introduces a new optimization for left semi joins with
runtime filters, allowing for a "direct return" path when certain
conditions are met. It also refactors runtime filter selectivity logic
to improve accuracy and maintainability, and adds mechanisms to detect
and propagate "IN" filters for more efficient query execution. The most
important changes are grouped below:
* Added a `left_semi_direct_return` flag to `HashJoinSharedState`,
enabling the left semi join to bypass hash table probing and directly
return results when a local IN filter is detected and only a single join
conjunct is present. This includes logic to detect such filters and
propagate the flag through the pipeline.
[[1]](diffhunk://#diff-9f27a64b1c3729dc2556ffa7ba55ae6b8f167671c759962eae9c1ce50f702296R625-R630)
[[2]](diffhunk://#diff-95f4d643dcceaebd86699edcee6c1bc3b920a4bffb3ea3162316666d18ddbc2aR242-R249)
[[3]](diffhunk://#diff-0732e01c1a3f38997ada381c43aff98286e86ca7519db5469a6e4dcdec5bce44R151-R158)
[[4]](diffhunk://#diff-c8eb587170d6e918998c8118d472105d8cebad1a5bb95445764c78e34dee2832R229-R234)
[[5]](diffhunk://#diff-c8eb587170d6e918998c8118d472105d8cebad1a5bb95445764c78e34dee2832L286-R296)
[[6]](diffhunk://#diff-86b1460edd261b4f708e89856c6d5bf3095a8856afa3852848c79f6d70247db4R62-R66)
[[7]](diffhunk://#diff-3110bab7d558f46b88ae1958b09ac369a92cac4bff98b280b2cf83d2d7aecbf4R211-R226)
[[8]](diffhunk://#diff-3110bab7d558f46b88ae1958b09ac369a92cac4bff98b280b2cf83d2d7aecbf4R812-R815)
* Replaced legacy selectivity tracking variables in `ColumnPredicate`
with a new `RuntimeFilterSelectivity` member, delegating all selectivity
logic (including resets, updates, and always_true checks) to this new
class for better encapsulation and flexibility.
[[1]](diffhunk://#diff-e9ad21f7dd14427a089c7540d93baef61756985194bd0c48a31cb3e9fee33c27L330-R333)
[[2]](diffhunk://#diff-e9ad21f7dd14427a089c7540d93baef61756985194bd0c48a31cb3e9fee33c27L390-R392)
[[3]](diffhunk://#diff-e9ad21f7dd14427a089c7540d93baef61756985194bd0c48a31cb3e9fee33c27L406-R418)
[[4]](diffhunk://#diff-e9ad21f7dd14427a089c7540d93baef61756985194bd0c48a31cb3e9fee33c27L442-R435)
[[5]](diffhunk://#diff-ac38473f151ccd99db4ed80dfbfa176f4f957ae2f5335ce8fbb4f5dbb0e932e9L377-R378)
* Refactored `RuntimeFilterSelectivity` to allow dynamic configuration
of sampling frequency, moved selectivity logic into instance methods,
and made the always-true logic depend on the configured sampling
frequency rather than a global config.
[[1]](diffhunk://#diff-dfe7d9e8512fd347fdf8a3cdc10e5399f703936b7ff4723acf5ae6ee2dc6dc85L37-R39)
[[2]](diffhunk://#diff-dfe7d9e8512fd347fdf8a3cdc10e5399f703936b7ff4723acf5ae6ee2dc6dc85L49-R49)
[[3]](diffhunk://#diff-dfe7d9e8512fd347fdf8a3cdc10e5399f703936b7ff4723acf5ae6ee2dc6dc85L64-R79)
[[4]](diffhunk://#diff-dfe7d9e8512fd347fdf8a3cdc10e5399f703936b7ff4723acf5ae6ee2dc6dc85L80-R92)
* Implemented detection of local IN filters in `RuntimeFilterProducer`
and `RuntimeFilterProducerHelper`, exposing methods to check for such
filters and using this information to enable the direct return
optimization.
[[1]](diffhunk://#diff-21ae533fb353c1ee2c03ee083e392b59f71e122af89b20b4767380aef83a6502R134-R141)
[[2]](diffhunk://#diff-bddbf8ad6f2ca00f5bd80c7ff32db5ec5d57a8ceab20a80b72e9296ec5281d7eR167-R175)
[[3]](diffhunk://#diff-8cf9a30c5ef28c70c390c5bf5696f184e2310bf1982172cc75e1942d9bbf656aR70-R71)
* Added a mechanism in the runtime filter consumer to set the
selectivity sampling frequency based on whether an IN filter was
detected, ensuring correct selectivity handling for these cases.
* Added state and methods in `RuntimeFilterWrapper` to track and expose
detection of IN filters.
[[1]](diffhunk://#diff-0115c52035ecbd3d57938f57ff167141a8aec665d90f386cfd44413b376dc322R89-R90)
[[2]](diffhunk://#diff-0115c52035ecbd3d57938f57ff167141a8aec665d90f386cfd44413b376dc322R125-R135)
* Changed the wait logic in `RuntimeFilterConsumer` to wait indefinitely
not only for bitmap filters but also when there is no remote target,
ensuring correctness in more scenarios.
* Removed an unnecessary include of `common/config.h` from
`runtime_filter_selectivity.h`.
BiteTheDDDDt added a commit that referenced this pull request Apr 14, 2026
…g VExprContext recreation (#62355)
introduced by #59832
Problem Summary:
RuntimeFilter selectivity tracking was completely non-functional because
`sampling_frequency` was lost during VExprContext recreation.
In `RuntimeFilterConsumer::_get_push_exprs()`, `sampling_frequency=32`
was set
on a temporary `probe_ctx` VExprContext. However, only
VRuntimeFilterWrapper
expressions (VExpr) were returned, not the VExprContext. When
`_append_rf_into_conjuncts()` later created a new VExprContext via
`VExprContext::create_shared(expr)`, the new context had default
`_sampling_frequency=-1` (DISABLE_SAMPLING).
With `_sampling_frequency=-1`, the condition `(_judge_counter++) >= -1`
evaluated to `0 >= -1 → true` on every call, causing
`reset_judge_selectivity()`
to fire every time. This meant selectivity counters were perpetually
reset
and never accumulated, making the runtime filter selectivity
optimization
completely ineffective.
**Fix**: Store `sampling_frequency` in VRuntimeFilterWrapper (which
survives
VExprContext recreation) and propagate it to VExprContext in
`VRuntimeFilterWrapper::open()`, which is called on both original and
cloned
contexts.
### Release note
Fixed a bug where RuntimeFilter selectivity tracking was non-functional
due to
sampling_frequency being lost during VExprContext recreation, causing
runtime
filters that should be skipped (due to low selectivity) to never be
identified.
### Check List (For Author)
- Test: Unit Test
- Added 2 regression tests to runtime_filter_selectivity_test.cpp
- Added 3 new tests in vruntimefilter_wrapper_sampling_test.cpp
- All 22 tests pass
- Behavior changed: No (selectivity tracking was broken before, this
makes it work as designed)
- Does this need documentation: No
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
morningman pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Apr 24, 2026
…g VExprContext recreation (apache#62355)
introduced by apache#59832
Problem Summary:
RuntimeFilter selectivity tracking was completely non-functional because
`sampling_frequency` was lost during VExprContext recreation.
In `RuntimeFilterConsumer::_get_push_exprs()`, `sampling_frequency=32`
was set
on a temporary `probe_ctx` VExprContext. However, only
VRuntimeFilterWrapper
expressions (VExpr) were returned, not the VExprContext. When
`_append_rf_into_conjuncts()` later created a new VExprContext via
`VExprContext::create_shared(expr)`, the new context had default
`_sampling_frequency=-1` (DISABLE_SAMPLING).
With `_sampling_frequency=-1`, the condition `(_judge_counter++) >= -1`
evaluated to `0 >= -1 → true` on every call, causing
`reset_judge_selectivity()`
to fire every time. This meant selectivity counters were perpetually
reset
and never accumulated, making the runtime filter selectivity
optimization
completely ineffective.
**Fix**: Store `sampling_frequency` in VRuntimeFilterWrapper (which
survives
VExprContext recreation) and propagate it to VExprContext in
`VRuntimeFilterWrapper::open()`, which is called on both original and
cloned
contexts.
Fixed a bug where RuntimeFilter selectivity tracking was non-functional
due to
sampling_frequency being lost during VExprContext recreation, causing
runtime
filters that should be skipped (due to low selectivity) to never be
identified.
- Test: Unit Test
- Added 2 regression tests to runtime_filter_selectivity_test.cpp
- Added 3 new tests in vruntimefilter_wrapper_sampling_test.cpp
- All 22 tests pass
- Behavior changed: No (selectivity tracking was broken before, this
makes it work as designed)
- Does this need documentation: No
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit 191061a)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by one committer.dev/4.0.4-mergedreviewedusercaseImportant user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@BiteTheDDDDt@hello-stephen@yiguolei@HappenLee