Uh oh!
There was an error while loading. Please reload this page.
[fix](nereids)Fix decompose repeat nest rewrite need to derive stats after construct cteProducer - #60811
Conversation
Thearas
commented
Feb 24, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
feiniaofeiafei
commented
Feb 24, 2026
run buildall |
doris-robot
commented
Feb 24, 2026
TPC-H: Total hot run time: 28882 ms |
doris-robot
commented
Feb 24, 2026
TPC-DS: Total hot run time: 184265 ms |
hello-stephen
commented
Feb 24, 2026
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Feb 24, 2026
FE Regression Coverage ReportIncrement line coverage |
feiniaofeiafei
commented
Feb 25, 2026
run cloud_p0 |
feiniaofeiafei
commented
Feb 25, 2026
run p0 |
hello-stephen
commented
Feb 25, 2026
FE Regression Coverage ReportIncrement line coverage |
feiniaofeiafei
commented
Feb 25, 2026
run cloud_p0 |
hello-stephen
commented
Feb 25, 2026
FE Regression Coverage ReportIncrement line coverage |
feiniaofeiafei
commented
Feb 27, 2026
run buildall |
doris-robot
commented
Feb 27, 2026
TPC-H: Total hot run time: 28879 ms |
doris-robot
commented
Feb 27, 2026
TPC-DS: Total hot run time: 184131 ms |
feiniaofeiafei
commented
Feb 27, 2026
run external |
feiniaofeiafei
commented
Feb 28, 2026
run buildall |
doris-robot
commented
Feb 28, 2026
TPC-H: Total hot run time: 28930 ms |
doris-robot
commented
Feb 28, 2026
TPC-DS: Total hot run time: 184500 ms |
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
…after construct cteProducer (apache#60811) Related PR: apache#59116 Reproduce: ```sql select a,b,c,c1 from ( select a,b,c,d,sum(d) c1 from t1 group by grouping sets((a,b,c),(a,b,c,d),(a),(a,b,c,c)) ) t group by rollup(a,b,c,c1); ``` Error: ```shell java.lang.IllegalArgumentException: Stats for CTE: CTEId#0 not found ``` Problem Summary: Root cause: This SQL triggers DecomposeRepeatWithPreAggregation twice: first for the inner LogicalAggregate, then for the outer LogicalAggregate. When rewriting the inner aggregate, a CTE structure is created (CTEAnchor, CTEProducer, CTEConsumer). When rewriting the outer aggregate, choosePreAggShuffleKeyPartitionExprs calls StatsDerive to derive statistics for the current subtree. At that point, the child of the outer LogicalAggregate is only the consumer subtree; the producer subtree is not part of the plan being visited. StatsDerive.visitLogicalCTEConsumer looks up producer statistics by CTEId, but the producer was never derived in this context, so the lookup fails with Stats for CTE: CTEId#0 not found. How to fix: In DecomposeRepeatWithPreAggregation.constructProducer(), derive statistics for the CTE producer when it is created. This ensures producer stats are stored in the context before the consumer is later visited. When StatsDerive visits LogicalCTEConsumer during the outer rewrite, it can find the producer stats correctly. Another fix: This pr increases the ndv requirement for the key when selecting a shuffle key. Only when the ndv > instance number * 512 will it be selected as the shuffle key. This is to avoid scenarios where low ndv leads to data skew.
…after construct cteProducer (apache#60811) Related PR: apache#59116 Reproduce: ```sql select a,b,c,c1 from ( select a,b,c,d,sum(d) c1 from t1 group by grouping sets((a,b,c),(a,b,c,d),(a),(a,b,c,c)) ) t group by rollup(a,b,c,c1); ``` Error: ```shell java.lang.IllegalArgumentException: Stats for CTE: CTEId#0 not found ``` Problem Summary: Root cause: This SQL triggers DecomposeRepeatWithPreAggregation twice: first for the inner LogicalAggregate, then for the outer LogicalAggregate. When rewriting the inner aggregate, a CTE structure is created (CTEAnchor, CTEProducer, CTEConsumer). When rewriting the outer aggregate, choosePreAggShuffleKeyPartitionExprs calls StatsDerive to derive statistics for the current subtree. At that point, the child of the outer LogicalAggregate is only the consumer subtree; the producer subtree is not part of the plan being visited. StatsDerive.visitLogicalCTEConsumer looks up producer statistics by CTEId, but the producer was never derived in this context, so the lookup fails with Stats for CTE: CTEId#0 not found. How to fix: In DecomposeRepeatWithPreAggregation.constructProducer(), derive statistics for the CTE producer when it is created. This ensures producer stats are stored in the context before the consumer is later visited. When StatsDerive visits LogicalCTEConsumer during the outer rewrite, it can find the producer stats correctly. Another fix: This pr increases the ndv requirement for the key when selecting a shuffle key. Only when the ndv > instance number * 512 will it be selected as the shuffle key. This is to avoid scenarios where low ndv leads to data skew.
What problem does this PR solve?
Related PR: #59116
Reproduce:
Error:
java.lang.IllegalArgumentException: Stats for CTE: CTEId#0 not foundProblem Summary:
Root cause:
This SQL triggers DecomposeRepeatWithPreAggregation twice: first for the inner LogicalAggregate, then for the outer LogicalAggregate.
When rewriting the inner aggregate, a CTE structure is created (CTEAnchor, CTEProducer, CTEConsumer).
When rewriting the outer aggregate, choosePreAggShuffleKeyPartitionExprs calls StatsDerive to derive statistics for the current subtree.
At that point, the child of the outer LogicalAggregate is only the consumer subtree; the producer subtree is not part of the plan being visited.
StatsDerive.visitLogicalCTEConsumer looks up producer statistics by CTEId, but the producer was never derived in this context, so the lookup fails with Stats for CTE: CTEId#0 not found.
How to fix:
In DecomposeRepeatWithPreAggregation.constructProducer(), derive statistics for the CTE producer when it is created. This ensures producer stats are stored in the context before the consumer is later visited. When StatsDerive visits LogicalCTEConsumer during the outer rewrite, it can find the producer stats correctly.
Another fix: This pr increases the ndv requirement for the key when selecting a shuffle key. Only when the ndv > instance number * 512 will it be selected as the shuffle key. This is to avoid scenarios where low ndv leads to data skew.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)