[fix](runtime-filter) Backport runtime filter fixes to branch-4.0 by BiteTheDDDDt · Pull Request #63083 · apache/doris

BiteTheDDDDt · 2026-05-08T11:03:35Z

What problem does this PR solve?

Issue Number: None

Problem Summary: Backport the branch-4.0 runtime filter fixes that are still missing from branch-4.0:

[fix](runtime-filter) Restore _applied_rf_num update in late arrival path #62872: restore _applied_rf_num = arrived_rf_num in the late-arrival runtime filter scanner path, so applied runtime filter count advances correctly.
[fix](be) Fix RuntimeFilter selectivity sampling_frequency lost during VExprContext recreation #62355: preserve runtime filter selectivity sampling_frequency across VRuntimeFilterWrapper / VExprContext recreation.

During the branch selection check, #62854 was not needed for branch-4.0/branch-4.1 because the affected length() OFFSET access-path optimization is absent there, and #60952 was already present on both branches.

Release note

None

Check List (For Author)

Test: Unit Test
- ./build.sh --be
- ./run-be-ut.sh --run --filter=ScannerLateArrivalRfTest.*:VRuntimeFilterWrapperSamplingTest.*:RuntimeFilterSelectivityTest.* -j 16
- build-support/check-format.sh
- build-support/run-clang-tidy.sh --base upstream/branch-4.0 --build-dir be/ut_build_ASAN via the current script from master, because branch-4.0 does not contain this helper script
Behavior changed: No
Does this need documentation: No

…path ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#59786 Problem Summary: PR apache#59786 (commit e78d089) refactored Scanner::try_append_late_arrival_runtime_filter() and accidentally removed the trailing `_applied_rf_num = arrived_rf_num;` assignment. As a result, _applied_rf_num is permanently 0 after construction: * the fast-path `_applied_rf_num == _total_rf_num` early return at the top of the function never fires, so every batch goes through the full late-arrival check; * `arrived_rf_num == _applied_rf_num` only short-circuits when no runtime filter has ever arrived, so once any RF arrives every subsequent call needlessly clears _conjuncts, re-clones them, and appends the old ctxs into _stale_expr_ctxs (CPU waste + slow memory growth); * the `ApplyAllRuntimeFilters=True` info string in profile (file_scanner.cpp) is never emitted; * `DCHECK(_applied_rf_num < _total_rf_num)` is effectively dead because the left-hand side is always 0. Restore the single missing assignment after cloning the new conjunct ctxs to bring back the original behavior. ### Release note None ### Check List (For Author) - Test: No need to test (one-line restoration of removed assignment; behavior covered by existing runtime-filter regression tests) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 884c978)

Related PR: apache#62872 Problem Summary: Lock in the fix for `Scanner::try_append_late_arrival_runtime_filter` so the regression introduced in PR apache#59786 cannot reappear. The new BE unit test instantiates a Scanner over MockScanLocalState/MockScanOperatorX, signals two runtime filters, and asserts that `_applied_rf_num` advances to `_total_rf_num` and that the second invocation hits the fast-path early return without re-cloning conjuncts. None - Test: Unit Test - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 675dee1)

…g VExprContext recreation Issue Number: close #xxx Problem Summary: RuntimeFilter selectivity tracking was completely non-functional because `sampling_frequency` was lost during VExprContext recreation. In `RuntimeFilterConsumer::_get_push_exprs()`, `sampling_frequency=32` was set on a temporary `probe_ctx` VExprContext. However, only VRuntimeFilterWrapper expressions (VExpr) were returned, not the VExprContext. When `_append_rf_into_conjuncts()` later created a new VExprContext via `VExprContext::create_shared(expr)`, the new context had default `_sampling_frequency=-1` (DISABLE_SAMPLING). With `_sampling_frequency=-1`, the condition `(_judge_counter++) >= -1` evaluated to `0 >= -1 → true` on every call, causing `reset_judge_selectivity()` to fire every time. This meant selectivity counters were perpetually reset and never accumulated, making the runtime filter selectivity optimization completely ineffective. The fix stores `sampling_frequency` in VRuntimeFilterWrapper (which survives VExprContext recreation) and propagates it to VExprContext in `VRuntimeFilterWrapper::open()`, which is called on both original and cloned contexts. Fixed a bug where RuntimeFilter selectivity tracking was non-functional due to sampling_frequency being lost during VExprContext recreation, causing runtime filters that should be skipped (due to low selectivity) to never be identified. - Test: Unit Test - Added 2 regression tests to runtime_filter_selectivity_test.cpp - Added 3 new tests in vruntimefilter_wrapper_sampling_test.cpp - All 22 tests pass - Behavior changed: No (selectivity tracking was broken before, this makes it work as designed) - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit f741922)

(cherry picked from commit 3c4d25e)

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> (cherry picked from commit 69e6004)

(cherry picked from commit f9a109b)

…ing test StubVExpr (leaf node with no children) defaulted to is_constant()=true via VExpr::is_constant()'s all_of on empty children. On second open() call (context2), _constant_col was already populated from context1, causing DCHECK(column_wrapper != nullptr) to fail in get_const_col() when called with nullptr from VExpr::open(). Fix: override is_constant() to return false, matching SLOT_REF semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 641c010)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

### What problem does this PR solve? Issue Number: closeapache#62872 Related PR: apache#62872 Problem Summary: Adapt the late-arrival runtime filter scanner unit test to branch-4.0 namespaces by qualifying DataTypeFactory. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=ScannerLateArrivalRfTest.*:VRuntimeFilterWrapperSamplingTest.*:RuntimeFilterSelectivityTest.* -j 16 - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hello-stephen · 2026-05-08T11:03:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

Backports two BE runtime-filter fixes into branch-4.0: (1) correctly advancing the scanner’s applied-RF counter in the late-arrival path, and (2) preserving runtime-filter selectivity sampling_frequency across VExprContext recreation by storing it on VRuntimeFilterWrapper and reapplying it during open().

Changes:

Restore _applied_rf_num = arrived_rf_num after late-arrival RF conjunct re-cloning in Scanner::try_append_late_arrival_runtime_filter().
Persist and propagate runtime-filter selectivity sampling_frequency by adding it to VRuntimeFilterWrapper and setting it on the (recreated) VExprContext during open().
Add/extend BE unit tests covering late-arrival RF counter advancement and selectivity sampling behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
be/src/vec/exec/scan/scanner.cpp	Restores `_applied_rf_num` update so late-arrival RF application state advances correctly.
be/src/runtime_filter/runtime_filter_consumer.cpp	Computes sampling frequency once and passes it into `VRuntimeFilterWrapper` so it survives context recreation.
be/src/vec/exprs/vruntimefilter_wrapper.h	Extends wrapper to store `sampling_frequency` (defaulting to DISABLE) in the expression object.
be/src/vec/exprs/vruntimefilter_wrapper.cpp	Propagates stored `sampling_frequency` to the active `VExprContext` in `open()`.
be/test/vec/exec/scanner_late_arrival_rf_test.cpp	New regression test verifying late-arrival path advances `_applied_rf_num` and fast-path short-circuits on subsequent calls.
be/test/runtime_filter/vruntimefilter_wrapper_sampling_test.cpp	New tests validating `sampling_frequency` propagation and survival across `VExprContext` recreation.
be/test/runtime_filter/runtime_filter_selectivity_test.cpp	Adds regression tests around default/valid sampling frequencies and reset/accumulation behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+ SlotDescriptor slot_desc;
+ slot_desc._type = vectorized::DataTypeFactory::instance().create_data_type(
+ PrimitiveType::TYPE_INT, false);
+ TupleDescriptor tuple_desc;


BiteTheDDDDt · 2026-05-08T11:34:25Z

run buildall

hello-stephen · 2026-05-08T14:38:49Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (11/11) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.43% (25474/35664)
Line Coverage	54.21% (269522/497221)
Region Coverage	51.70% (222734/430860)
Branch Coverage	53.17% (95966/180485)

BiteTheDDDDtand others added 9 commits May 8, 2026 16:55

format
61d76d0
(cherry picked from commit 3c4d25e)

Potential fix for pull request finding
d0e801a
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> (cherry picked from commit 69e6004)

update
37bdfbd
(cherry picked from commit f9a109b)

[chore](test) Format branch-4.0 scanner runtime filter test
41f9c51
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

CopilotAI review requested due to automatic review settings May 8, 2026 11:03

BiteTheDDDDt requested review from morningman and yiguolei as code owners May 8, 2026 11:03

Copilotstarted reviewing on behalf of BiteTheDDDDt May 8, 2026 11:04 View session

CopilotAI reviewed May 8, 2026
View reviewed changes

Comment threadbe/test/vec/exec/scanner_late_arrival_rf_test.cpp
SlotDescriptor slot_desc;
slot_desc._type = vectorized::DataTypeFactory::instance().create_data_type(
PrimitiveType::TYPE_INT, false);
TupleDescriptor tuple_desc;

yiguolei approved these changes May 11, 2026
View reviewed changes

yiguolei merged commit 89b31ef into apache:branch-4.0May 11, 2026
33 of 36 checks passed

Navigation Menu

Uh oh!

[fix](runtime-filter) Backport runtime filter fixes to branch-4.0 - #63083

[fix](runtime-filter) Backport runtime filter fixes to branch-4.0#63083
yiguolei merged 9 commits into
apache:branch-4.0from
BiteTheDDDDt:pick-62872-62355-branch-4.0

BiteTheDDDDt commented May 8, 2026

Uh oh!

hello-stephen commented May 8, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

BiteTheDDDDt commented May 8, 2026

Uh oh!

hello-stephen commented May 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

BiteTheDDDDt commented May 8, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented May 8, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

BiteTheDDDDt commented May 8, 2026

Uh oh!

hello-stephen commented May 8, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants