Uh oh!
There was an error while loading. Please reload this page.
[Feat](nereids) support pull up predicate from set operator - #39450
Merged
Conversation
doris-robot
commented
Aug 15, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
feiniaofeiafei
commented
Aug 15, 2024
CollaboratorAuthor
run buildall |
doris-robot
commented
Aug 15, 2024
TPC-H: Total hot run time: 38093 ms |
doris-robot
commented
Aug 15, 2024
TPC-DS: Total hot run time: 189870 ms |
doris-robot
commented
Aug 15, 2024
ClickBench: Total hot run time: 31.37 s |
feiniaofeiafeiforce-pushed
the
pull_up_from_set_op
branch
from
August 16, 2024 03:25
62de7b3 to
d2e8ab6Comparefeiniaofeiafei
commented
Aug 16, 2024
CollaboratorAuthor
run buildall |
doris-robot
commented
Aug 16, 2024
TPC-H: Total hot run time: 38116 ms |
doris-robot
commented
Aug 16, 2024
TPC-DS: Total hot run time: 195770 ms |
doris-robot
commented
Aug 16, 2024
ClickBench: Total hot run time: 31.75 s |
xzj7019
reviewed
Aug 16, 2024
| custom(RuleType.ELIMINATE_SORT, EliminateSort::new), | ||
| bottomUp(new EliminateEmptyRelation()) | ||
| bottomUp(new EliminateEmptyRelation()), | ||
| topDown(new PushProjectIntoUnion()), |
feiniaofeiafei
commented
Aug 16, 2024
CollaboratorAuthor
run buildall |
doris-robot
commented
Aug 16, 2024
TPC-H: Total hot run time: 38044 ms |
feiniaofeiafei
commented
Aug 16, 2024
CollaboratorAuthor
run buildall |
doris-robot
commented
Aug 16, 2024
TPC-H: Total hot run time: 38182 ms |
doris-robot
commented
Aug 16, 2024
TPC-DS: Total hot run time: 196994 ms |
doris-robot
commented
Aug 16, 2024
ClickBench: Total hot run time: 31.05 s |
feiniaofeiafei
commented
Aug 17, 2024
CollaboratorAuthor
run buildall |
doris-robot
commented
Aug 17, 2024
TPC-H: Total hot run time: 38062 ms |
doris-robot
commented
Aug 17, 2024
TPC-DS: Total hot run time: 195919 ms |
doris-robot
commented
Aug 17, 2024
ClickBench: Total hot run time: 31.17 s |
feiniaofeiafei
commented
Aug 18, 2024
CollaboratorAuthor
run p0 |
feiniaofeiafei
commented
Aug 18, 2024
CollaboratorAuthor
run buildall |
doris-robot
commented
Aug 18, 2024
TPC-H: Total hot run time: 38427 ms |
doris-robot
commented
Aug 18, 2024
TPC-DS: Total hot run time: 190409 ms |
doris-robot
commented
Aug 18, 2024
ClickBench: Total hot run time: 32.2 s |
feiniaofeiafei
commented
Aug 19, 2024
CollaboratorAuthor
run p0 |
feiniaofeiafei
commented
Aug 19, 2024
CollaboratorAuthor
run cloud_p1 |
feiniaofeiafei
commented
Aug 19, 2024
CollaboratorAuthor
run buildall |
doris-robot
commented
Aug 19, 2024
TPC-H: Total hot run time: 38629 ms |
feiniaofeiafei added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Oct 15, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
924060929 pushed a commit
that referenced
this pull request
Oct 24, 2024
…41908) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
924060929 pushed a commit
that referenced
this pull request
Oct 24, 2024
…on all (#41613) (#41909) introduce by #31811 and #39450 ```sql select count(1) from(select 3, 6 union all select 1, 3) t ``` wrong LogicalUnion plan: ```sql LogicalUnion( qualifier=ALL, outputs=[3#6], regularChildrenOutputs=[], constantExprsList=[[], []], hasPushedFilter=false ``` this sql will report error in explain, because the logical union outputs has a slot, but the logical union has no child and has a empty constantExprList, which is wrong set in column prune. this pr fixes it by consider when require columns is empty and keep the min slot and min slot corresponding const expressions.
feiniaofeiafei added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 15, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
feiniaofeiafei added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 21, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
feiniaofeiafei added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 21, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
feiniaofeiafei added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 21, 2024
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
morrySnow pushed a commit
that referenced
this pull request
Dec 3, 2024
Closed
morrySnow pushed a commit
that referenced
this pull request
Apr 13, 2026
…62299) ### What problem does this PR solve? Problem Summary: introduced by #39450 `PullUpPredicates.getFiltersFromUnionConstExprs()` incorrectly removes `NullLiteral` from UNION ALL constant expressions without compensating with `OR IS NULL`, producing invalid pull-up predicates. For example, `SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL` generates the pull-up predicate `n IN (1, 3)`, which is NOT true for the NULL row (`NULL IN (1, 3)` evaluates to NULL, not TRUE). When `InferPredicates` pushes this predicate from one INTERSECT child to another, the IN filter incorrectly eliminates NULL rows. Since INTERSECT treats NULL=NULL per SQL standard, this causes incorrect query results with missing NULLs. **Root Cause**: In `getFiltersFromUnionConstExprs()`, line 467: ```java options.removeIf(option -> option instanceof NullLiteral); ``` removes NULL but does not add `OR IS NULL` to compensate. **Fix**: When NULL constants are present, generate `n IN (...) OR n IS NULL` instead of just `n IN (...)`, making the predicate valid for all rows including NULL. **Reproducing Query**: ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) ( SELECT (n * 2) AS n FROM tbl0 INTERSECT SELECT (n * 2) AS n FROM tbl1 ); -- Expected: {2, NULL} Actual before fix: {2} ``` ### Release note Fixed a bug where INTERSECT and EXCEPT queries could incorrectly drop NULL rows when the optimizer inferred and pushed down predicates from UNION ALL constant expressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actionsBot
pushed a commit
that referenced
this pull request
Apr 13, 2026
…62299) ### What problem does this PR solve? Problem Summary: introduced by #39450 `PullUpPredicates.getFiltersFromUnionConstExprs()` incorrectly removes `NullLiteral` from UNION ALL constant expressions without compensating with `OR IS NULL`, producing invalid pull-up predicates. For example, `SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL` generates the pull-up predicate `n IN (1, 3)`, which is NOT true for the NULL row (`NULL IN (1, 3)` evaluates to NULL, not TRUE). When `InferPredicates` pushes this predicate from one INTERSECT child to another, the IN filter incorrectly eliminates NULL rows. Since INTERSECT treats NULL=NULL per SQL standard, this causes incorrect query results with missing NULLs. **Root Cause**: In `getFiltersFromUnionConstExprs()`, line 467: ```java options.removeIf(option -> option instanceof NullLiteral); ``` removes NULL but does not add `OR IS NULL` to compensate. **Fix**: When NULL constants are present, generate `n IN (...) OR n IS NULL` instead of just `n IN (...)`, making the predicate valid for all rows including NULL. **Reproducing Query**: ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) ( SELECT (n * 2) AS n FROM tbl0 INTERSECT SELECT (n * 2) AS n FROM tbl1 ); -- Expected: {2, NULL} Actual before fix: {2} ``` ### Release note Fixed a bug where INTERSECT and EXCEPT queries could incorrectly drop NULL rows when the optimizer inferred and pushed down predicates from UNION ALL constant expressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actionsBot
pushed a commit
that referenced
this pull request
Apr 13, 2026
…62299) ### What problem does this PR solve? Problem Summary: introduced by #39450 `PullUpPredicates.getFiltersFromUnionConstExprs()` incorrectly removes `NullLiteral` from UNION ALL constant expressions without compensating with `OR IS NULL`, producing invalid pull-up predicates. For example, `SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL` generates the pull-up predicate `n IN (1, 3)`, which is NOT true for the NULL row (`NULL IN (1, 3)` evaluates to NULL, not TRUE). When `InferPredicates` pushes this predicate from one INTERSECT child to another, the IN filter incorrectly eliminates NULL rows. Since INTERSECT treats NULL=NULL per SQL standard, this causes incorrect query results with missing NULLs. **Root Cause**: In `getFiltersFromUnionConstExprs()`, line 467: ```java options.removeIf(option -> option instanceof NullLiteral); ``` removes NULL but does not add `OR IS NULL` to compensate. **Fix**: When NULL constants are present, generate `n IN (...) OR n IS NULL` instead of just `n IN (...)`, making the predicate valid for all rows including NULL. **Reproducing Query**: ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) ( SELECT (n * 2) AS n FROM tbl0 INTERSECT SELECT (n * 2) AS n FROM tbl1 ); -- Expected: {2, NULL} Actual before fix: {2} ``` ### Release note Fixed a bug where INTERSECT and EXCEPT queries could incorrectly drop NULL rows when the optimizer inferred and pushed down predicates from UNION ALL constant expressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…9450) This pr complete the support scope of PullUpPredicate: select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1 select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1 select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1 select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa') And support infer predicate for except and intersect: select a,b from test1 where a<1 except all select a,b from test2 where b<1; -> select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1; select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; -> select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;
yujun777 added a commit
to yujun777/doris
that referenced
this pull request
Apr 27, 2026
…-into-Union rules ### What problem does this PR solve? Issue Number: closeapache#62294 Problem Summary: When PushProjectIntoUnion folds a parent Project's expression into a LogicalUnion's constantExprsList, the outer Alias of a non-SlotReference project expression is preserved by ExpressionUtils.replaceNameExpression. Its ExprId then collides with the new UNION output ExprId (which comes from the parent project's output Alias) and is reused across every constant row of the same column. Downstream rules such as PushDownFilterThroughSetOperation rely on the invariant that each constantExprsList row carries NamedExpressions whose ExprIds are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results for queries like: WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; PushProjectThroughUnion has the same class of bug in its mixed-union constant-row branch: when MergeOneRowRelationIntoUnion folds a LogicalOneRowRelation into the union's constantExprsList, an outer Alias(Cast(slot)) project survives `outerProject.rewriteUp` with its parent ExprId intact, and then the same ExprId is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - PushProjectIntoUnion: after folding the substituted expression, re-wrap the result in a fresh Alias so each constant cell receives a new unique ExprId. Qualifier and name from the folded Alias are preserved. The SlotReference branch is left unchanged because it already returns the original NamedExpression from constExprs whose ExprIds are row-distinct and not equal to the UNION output ExprId. - PushProjectThroughUnion: same treatment for the constant-row non-Slot branch — re-wrap the rewritten Alias to allocate a fresh ExprId. Introduced by apache#39450 (closest commit touching the relevant lines; the file PushProjectIntoUnion was originally added by apache#27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as ((col*2) AS col), and harden the parallel PushProjectThroughUnion rule against the same class of ExprId collision. ### Check List (For Author) - Test: - Regression test: regression-test/suites/query_p0/set_operations/set_operation_exprid_reuse.groovy - Unit Test: fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushProjectIntoUnionTest.java - Unit Test: fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushProjectThroughUnionTest.java - Behavior changed: Yes (fixes wrong results for the PushProjectIntoUnion patterns above; PushProjectThroughUnion change preserves invariants without observable behavior change today) - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
yujun777 added a commit
to yujun777/doris
that referenced
this pull request
May 5, 2026
…-into-Union rules ### What problem does this PR solve? Issue Number: closeapache#62294 Problem Summary: When PushProjectIntoUnion folds a parent Project's expression into a LogicalUnion's constantExprsList, the outer Alias of a non-SlotReference project expression is preserved by ExpressionUtils.replaceNameExpression. Its ExprId then collides with the new UNION output ExprId (which comes from the parent project's output Alias) and is reused across every constant row of the same column. Downstream rules such as PushDownFilterThroughSetOperation rely on the invariant that each constantExprsList row carries NamedExpressions whose ExprIds are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results for queries like: WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; PushProjectThroughUnion has the same class of bug in its mixed-union constant-row branch: when MergeOneRowRelationIntoUnion folds a LogicalOneRowRelation into the union's constantExprsList, an outer Alias(Cast(slot)) project survives `outerProject.rewriteUp` with its parent ExprId intact, and then the same ExprId is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - PushProjectIntoUnion: after folding the substituted expression, re-wrap the result in a fresh Alias so each constant cell receives a new unique ExprId. Qualifier and name from the folded Alias are preserved. The SlotReference branch is left unchanged because it already returns the original NamedExpression from constExprs whose ExprIds are row-distinct and not equal to the UNION output ExprId. - PushProjectThroughUnion: same treatment for the constant-row non-Slot branch — re-wrap the rewritten Alias to allocate a fresh ExprId. Introduced by apache#39450 (closest commit touching the relevant lines; the file PushProjectIntoUnion was originally added by apache#27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as ((col*2) AS col), and harden the parallel PushProjectThroughUnion rule against the same class of ExprId collision. ### Check List (For Author) - Test: - Regression test: regression-test/suites/query_p0/set_operations/set_operation_exprid_reuse.groovy - Unit Test: fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushProjectIntoUnionTest.java - Unit Test: fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushProjectThroughUnionTest.java - Behavior changed: Yes (fixes wrong results for the PushProjectIntoUnion patterns above; PushProjectThroughUnion change preserves invariants without observable behavior change today) - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
yujun777 added a commit
to yujun777/doris
that referenced
this pull request
May 5, 2026
…-into-Union rules ### What problem does this PR solve? Issue Number: closeapache#62294 Problem Summary: When PushProjectIntoUnion folds a parent Project's expression into a LogicalUnion's constantExprsList, the outer Alias of a non-SlotReference project expression is preserved by ExpressionUtils.replaceNameExpression. Its ExprId then collides with the new UNION output ExprId (which comes from the parent project's output Alias) and is reused across every constant row of the same column. Downstream rules such as PushDownFilterThroughSetOperation rely on the invariant that each constantExprsList row carries NamedExpressions whose ExprIds are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results for queries like: WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; PushProjectThroughUnion has the same class of bug in its mixed-union constant-row branch: when MergeOneRowRelationIntoUnion folds a LogicalOneRowRelation into the union's constantExprsList, an outer Alias(Cast(slot)) project survives `outerProject.rewriteUp` with its parent ExprId intact, and then the same ExprId is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - PushProjectIntoUnion: after folding the substituted expression, re-wrap the result in a fresh Alias so each constant cell receives a new unique ExprId. Qualifier and name from the folded Alias are preserved. The SlotReference branch is left unchanged because it already returns the original NamedExpression from constExprs whose ExprIds are row-distinct and not equal to the UNION output ExprId. - PushProjectThroughUnion: same treatment for the constant-row non-Slot branch — re-wrap the rewritten Alias to allocate a fresh ExprId. Introduced by apache#39450 (closest commit touching the relevant lines; the file PushProjectIntoUnion was originally added by apache#27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as ((col*2) AS col), and harden the parallel PushProjectThroughUnion rule against the same class of ExprId collision. ### Check List (For Author) - Test: - Regression test: regression-test/suites/query_p0/set_operations/set_operation_exprid_reuse.groovy - Unit Test: fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushProjectIntoUnionTest.java - Unit Test: fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushProjectThroughUnionTest.java - Behavior changed: Yes (fixes wrong results for the PushProjectIntoUnion patterns above; PushProjectThroughUnion change preserves invariants without observable behavior change today) - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
morrySnow pushed a commit
that referenced
this pull request
May 6, 2026
…ct into Union (#62296) ### What problem does this PR solve? Issue Number: close#62294 Problem Summary: When `PushProjectIntoUnion` folds a parent `Project`'s expression into a `LogicalUnion`'s `constantExprsList`, the outer `Alias` of a non-`SlotReference` project expression is preserved by `ExpressionUtils.replaceNameExpression`. Its `ExprId` then collides with the new UNION output `ExprId` (which comes from the parent project's output `Alias`) and is reused across every constant row of the same column. Downstream rules such as `PushDownFilterThroughSetOperation` rely on the invariant that each `constantExprsList` row carries `NamedExpression`s whose `ExprId`s are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results, e.g. ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; ``` `PushProjectThroughUnion` has the same class of bug in its mixed-union constant-row branch: when `MergeOneRowRelationIntoUnion` folds a `LogicalOneRowRelation` into the union's `constantExprsList`, an outer `Alias(Cast(slot))` project survives `outerProject.rewriteUp` with its parent `ExprId` intact, and then the same `ExprId` is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - `PushProjectIntoUnion`: after folding the substituted expression, re-wrap the result in a fresh `Alias` so each constant cell receives a new unique `ExprId`. Qualifier and name from the folded `Alias` are preserved. The `SlotReference` branch is left unchanged because it already returns the original `NamedExpression` from `constExprs` whose `ExprId`s are row-distinct and not equal to the UNION output `ExprId`. - `PushProjectThroughUnion`: same treatment for the constant-row non-Slot branch — re-wrap the rewritten `Alias` to allocate a fresh `ExprId`. Introduced by #39450 (closest commit touching the relevant lines; the file `PushProjectIntoUnion` was originally added by #27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as `((col*2) AS col)`, and harden the parallel `PushProjectThroughUnion` rule against the same class of ExprId collision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actionsBot
pushed a commit
that referenced
this pull request
May 6, 2026
…ct into Union (#62296) ### What problem does this PR solve? Issue Number: close#62294 Problem Summary: When `PushProjectIntoUnion` folds a parent `Project`'s expression into a `LogicalUnion`'s `constantExprsList`, the outer `Alias` of a non-`SlotReference` project expression is preserved by `ExpressionUtils.replaceNameExpression`. Its `ExprId` then collides with the new UNION output `ExprId` (which comes from the parent project's output `Alias`) and is reused across every constant row of the same column. Downstream rules such as `PushDownFilterThroughSetOperation` rely on the invariant that each `constantExprsList` row carries `NamedExpression`s whose `ExprId`s are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results, e.g. ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; ``` `PushProjectThroughUnion` has the same class of bug in its mixed-union constant-row branch: when `MergeOneRowRelationIntoUnion` folds a `LogicalOneRowRelation` into the union's `constantExprsList`, an outer `Alias(Cast(slot))` project survives `outerProject.rewriteUp` with its parent `ExprId` intact, and then the same `ExprId` is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - `PushProjectIntoUnion`: after folding the substituted expression, re-wrap the result in a fresh `Alias` so each constant cell receives a new unique `ExprId`. Qualifier and name from the folded `Alias` are preserved. The `SlotReference` branch is left unchanged because it already returns the original `NamedExpression` from `constExprs` whose `ExprId`s are row-distinct and not equal to the UNION output `ExprId`. - `PushProjectThroughUnion`: same treatment for the constant-row non-Slot branch — re-wrap the rewritten `Alias` to allocate a fresh `ExprId`. Introduced by #39450 (closest commit touching the relevant lines; the file `PushProjectIntoUnion` was originally added by #27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as `((col*2) AS col)`, and harden the parallel `PushProjectThroughUnion` rule against the same class of ExprId collision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actionsBot
pushed a commit
that referenced
this pull request
May 6, 2026
…ct into Union (#62296) ### What problem does this PR solve? Issue Number: close#62294 Problem Summary: When `PushProjectIntoUnion` folds a parent `Project`'s expression into a `LogicalUnion`'s `constantExprsList`, the outer `Alias` of a non-`SlotReference` project expression is preserved by `ExpressionUtils.replaceNameExpression`. Its `ExprId` then collides with the new UNION output `ExprId` (which comes from the parent project's output `Alias`) and is reused across every constant row of the same column. Downstream rules such as `PushDownFilterThroughSetOperation` rely on the invariant that each `constantExprsList` row carries `NamedExpression`s whose `ExprId`s are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results, e.g. ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; ``` `PushProjectThroughUnion` has the same class of bug in its mixed-union constant-row branch: when `MergeOneRowRelationIntoUnion` folds a `LogicalOneRowRelation` into the union's `constantExprsList`, an outer `Alias(Cast(slot))` project survives `outerProject.rewriteUp` with its parent `ExprId` intact, and then the same `ExprId` is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - `PushProjectIntoUnion`: after folding the substituted expression, re-wrap the result in a fresh `Alias` so each constant cell receives a new unique `ExprId`. Qualifier and name from the folded `Alias` are preserved. The `SlotReference` branch is left unchanged because it already returns the original `NamedExpression` from `constExprs` whose `ExprId`s are row-distinct and not equal to the UNION output `ExprId`. - `PushProjectThroughUnion`: same treatment for the constant-row non-Slot branch — re-wrap the rewritten `Alias` to allocate a fresh `ExprId`. Introduced by #39450 (closest commit touching the relevant lines; the file `PushProjectIntoUnion` was originally added by #27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as `((col*2) AS col)`, and harden the parallel `PushProjectThroughUnion` rule against the same class of ExprId collision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
yiguolei pushed a commit
that referenced
this pull request
May 7, 2026
…ct into Union (#62296) ### What problem does this PR solve? Issue Number: close#62294 Problem Summary: When `PushProjectIntoUnion` folds a parent `Project`'s expression into a `LogicalUnion`'s `constantExprsList`, the outer `Alias` of a non-`SlotReference` project expression is preserved by `ExpressionUtils.replaceNameExpression`. Its `ExprId` then collides with the new UNION output `ExprId` (which comes from the parent project's output `Alias`) and is reused across every constant row of the same column. Downstream rules such as `PushDownFilterThroughSetOperation` rely on the invariant that each `constantExprsList` row carries `NamedExpression`s whose `ExprId`s are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results, e.g. ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; ``` `PushProjectThroughUnion` has the same class of bug in its mixed-union constant-row branch: when `MergeOneRowRelationIntoUnion` folds a `LogicalOneRowRelation` into the union's `constantExprsList`, an outer `Alias(Cast(slot))` project survives `outerProject.rewriteUp` with its parent `ExprId` intact, and then the same `ExprId` is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - `PushProjectIntoUnion`: after folding the substituted expression, re-wrap the result in a fresh `Alias` so each constant cell receives a new unique `ExprId`. Qualifier and name from the folded `Alias` are preserved. The `SlotReference` branch is left unchanged because it already returns the original `NamedExpression` from `constExprs` whose `ExprId`s are row-distinct and not equal to the UNION output `ExprId`. - `PushProjectThroughUnion`: same treatment for the constant-row non-Slot branch — re-wrap the rewritten `Alias` to allocate a fresh `ExprId`. Introduced by #39450 (closest commit touching the relevant lines; the file `PushProjectIntoUnion` was originally added by #27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as `((col*2) AS col)`, and harden the parallel `PushProjectThroughUnion` rule against the same class of ExprId collision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
zhaorongsheng pushed a commit
to zhaorongsheng/doris
that referenced
this pull request
Jun 4, 2026
…ct into Union (apache#62296) ### What problem does this PR solve? Issue Number: closeapache#62294 Problem Summary: When `PushProjectIntoUnion` folds a parent `Project`'s expression into a `LogicalUnion`'s `constantExprsList`, the outer `Alias` of a non-`SlotReference` project expression is preserved by `ExpressionUtils.replaceNameExpression`. Its `ExprId` then collides with the new UNION output `ExprId` (which comes from the parent project's output `Alias`) and is reused across every constant row of the same column. Downstream rules such as `PushDownFilterThroughSetOperation` rely on the invariant that each `constantExprsList` row carries `NamedExpression`s whose `ExprId`s are distinct from the UNION output and from each other across rows; the collision causes them to mis-rewrite the plan and return wrong results, e.g. ```sql WITH tbl0(n) AS (SELECT 1 UNION ALL SELECT 3 UNION ALL SELECT NULL), tbl1(n) AS (SELECT 2 UNION ALL SELECT NULL UNION ALL SELECT 1) SELECT (n*2) AS n FROM tbl0 INTERSECT SELECT (n*2) AS n FROM tbl1; ``` `PushProjectThroughUnion` has the same class of bug in its mixed-union constant-row branch: when `MergeOneRowRelationIntoUnion` folds a `LogicalOneRowRelation` into the union's `constantExprsList`, an outer `Alias(Cast(slot))` project survives `outerProject.rewriteUp` with its parent `ExprId` intact, and then the same `ExprId` is reused as the new UNION output via `project.toSlot()`. The plan-level invariant is broken even though no current downstream rule appears to mis-handle the specific shape today. Fix: - `PushProjectIntoUnion`: after folding the substituted expression, re-wrap the result in a fresh `Alias` so each constant cell receives a new unique `ExprId`. Qualifier and name from the folded `Alias` are preserved. The `SlotReference` branch is left unchanged because it already returns the original `NamedExpression` from `constExprs` whose `ExprId`s are row-distinct and not equal to the UNION output `ExprId`. - `PushProjectThroughUnion`: same treatment for the constant-row non-Slot branch — re-wrap the rewritten `Alias` to allocate a fresh `ExprId`. Introduced by apache#39450 (closest commit touching the relevant lines; the file `PushProjectIntoUnion` was originally added by apache#27947). ### Release note Fix wrong results for INTERSECT/EXCEPT/UNION over constant rows when the projection contains expressions such as `((col*2) AS col)`, and harden the parallel `PushProjectThroughUnion` rule against the same class of ExprId collision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pr complete the support scope of PullUpPredicate:
select a,b from test1 where a<1 union all select a,b from test2 where a<1; pull up filter a<1
select a,b from test1 where a<1 intersect all select a,b from test2 where b<1; pull up filter a<1 and b<1
select a,b from test1 where a<1 except all select a,b from test2 where b<1; pull up filter a<1
select 1 a, 'bbb' b union (all) select 2,'aa'; pull up filter a in(1,2) and b in ('bbb','aa')
And support infer predicate for except and intersect:
select a,b from test1 where a<1 except all select a,b from test2 where b<1;
->
select a,b from test1 where a<1 except all select a,b from test2 where b<1 and a<1;
select a,b from test1 where a<1 intersect all select a,b from test2 where b<1;
->
select a,b from test1 where a<1 and b < 1 intersect all select a,b from test2 where b<1 and a < 1;