Uh oh!
There was an error while loading. Please reload this page.
branch-4.0: [fix](nereids) Guard UniqueFunction in multiple filter/topn pushdown rules #62742 - #65259
Merged
Merged
Conversation
…rules (apache#62742) Problem Summary: Several Nereids rewrite rules still moved predicates containing non-idempotent (unique) functions such as rand() / uuid() / random_bytes() across operator boundaries in ways that changed query semantics. The common root cause is that a predicate like `rand() > 0.5` has an empty input-slot set, so the `containsAll(emptySet)` / `allMatch` guards used by these rules silently returned true and allowed unsafe push down / elimination. This PR adds `containsUniqueFunction()` guards to the following rules: 1. PushDownFilterThroughRepeat: skip conjuncts with unique functions. Pushing `rand() > x` below Repeat changes which rows feed each grouping set and alters aggregate results. 2. PushDownFilterThroughWindow: skip conjuncts with unique functions. Pushing a unique predicate below a window operator re-samples the base rows and changes every window-function value. 3. PushDownFilterThroughPartitionTopN: same as Window - skip unique conjuncts in the split loop. 4. PushDownFilterThroughSetOperation: do not push volatile conjuncts below `UNION DISTINCT`, `INTERSECT`, or `EXCEPT`; only `UNION ALL` keeps the original row-to-row semantics. 5. PushDownJoinOtherCondition: keep volatile ON predicates in the join when pushing them into a single child would change evaluation from per joined pair to per input row. 6. AddProjectForVolatileExpression: when a volatile expression is repeated after rewrites such as BETWEEN expansion, materialize it via a child project so repeated references share one value instead of being re-evaluated independently. 7. InferPredicates / JoinUtils.isHashJoinCondition: add the same unique function guards to avoid deriving or classifying unsafe predicates. Fix wrong results when predicates containing rand(), uuid(), random_bytes(), or uuid_numeric() are pushed across Repeat, Window, PartitionTopN, SetOperation, or Join boundaries, and when repeated volatile expressions are materialized for reuse. --------- Co-authored-by: yujun777 <yujun777@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 8255f94)
Backport the unique-function guard changes from apache#62742 onto branch-4.0 and align the affected rewrite rules with 4.0-specific unique-function semantics. Key changes: - replace the unsupported volatile-expression checks in the touched 4.0 rewrite paths with unique-function checks - adapt AddProjectForUniqueFunction join rewriting to use UniqueFunction in 4.0 - keep PushDownFilterThroughSetOperation conservative on 4.0 by not pushing unique-function predicates through set operations, including UNION ALL - update the affected regression suite/output for the 4.0 set-operation behavior Unit Test: - FE UT: AddProjectForUniqueFunctionTest, FindHashConditionForJoinTest, PushDownFilterThroughWindowTest, PushDownJoinOtherConditionTest, PushDownFilterThroughSetOperationTest, ProjectOtherJoinConditionForNestedLoopJoinTest, InferPredicatesTest, JoinUtilsTest - Regression: regression-test/suites/nereids_rules_p0/unique_function with -forceGenOut - Regression: regression-test/suites/nereids_rules_p0/infer_predicate/extend_infer_equal_predicate.groovy with -forceGenOut
yujun777
commented
Jul 6, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
Jul 6, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yujun777
commented
Jul 6, 2026
ContributorAuthor
run buildall |
yujun777
commented
Jul 6, 2026
ContributorAuthor
run cloud_p0 |
morningman
approved these changes
Jul 23, 2026
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-pick: #62742