Skip to content

branch-4.1: [feature](fe) Push down limit into CTE producer (#63675) - #64003

Merged
yiguolei merged 3 commits into
branch-4.1from
branch-4.1-63675
Aug 25, 2026
Merged

branch-4.1: [feature](fe) Push down limit into CTE producer (#63675)#64003
yiguolei merged 3 commits into
branch-4.1from
branch-4.1-63675

Conversation

@CalvinKirs

@CalvinKirsCalvinKirs commented Jun 2, 2026

Copy link
Copy Markdown
Member

Master PR: #63675

@CalvinKirs
[feature](fe) Push down limit into CTE producer (#63675)
5acefff
This PR adds CTE producer-side limit pushdown in Nereids.
When all CTE consumers only need a bounded number of rows, the optimizer
collects the required row count from each consumer, takes
the maximum value, and pushes that limit into the CTE producer. The
original consumer-side limit is still kept.
The rule only handles safe shapes:
```text
LogicalLimit
LogicalCTEConsumer
```
```text
LogicalLimit
LogicalProject
LogicalCTEConsumer
```
The project must be row-preserving.
## Scenarios
### 1. Direct Limit
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
LIMIT 10;
```
The consumer only needs 10 rows, so the CTE producer can produce at most
10 rows.
### 2. Project + Limit
```sql
WITH cte AS (
SELECT order_id, total_price, user_id FROM orders
)
SELECT order_id, total_price
FROM cte
LIMIT 10;
```
A normal project only prunes columns and does not change row count, so
the producer can still be limited to 10 rows.
### 3. Multiple Consumers + Limit
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte LIMIT 10
UNION ALL
SELECT * FROM cte LIMIT 20;
```
For multiple CTE consumers, the producer limit is:
```text
producerLimit = max(consumerLimit1, consumerLimit2, ...)
```
In this case, the pushed producer limit is 20.
If any consumer needs full CTE data, pushdown is skipped:
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte LIMIT 10
UNION ALL
SELECT * FROM cte;
```
### 4. Limit + Offset
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
LIMIT 10 OFFSET 100;
```
The consumer needs to skip 100 rows and then return 10 rows, so the
producer must provide at least 110 rows.
The producer side only truncates rows and does not apply offset:
```text
producerLimit = limit + offset
producerOffset = 0
```
### 5. SplitLimit
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
LIMIT 10 OFFSET 100;
```
Doris may split this into local/global limits. The local limit closest
to the CTE consumer already represents `limit + offset`.
The collector uses the local limit value directly and does not add
offset again.
### 6. Filter + Limit Is Not Matched
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
WHERE order_id > 10
LIMIT 10;
```
Filter can reduce rows before limit, so the producer may need more than
10 input rows. This rule does not push limit through filter.
### 7. TopN Is Not Matched
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT * FROM cte
ORDER BY order_id
LIMIT 10;
```
`ORDER BY ... LIMIT` is TopN. It needs the first N rows after ordering,
so it cannot be treated as a normal limit.
### 8. Join / Aggregate / Window / Sort Are Not Matched
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT *
FROM cte JOIN users ON cte.user_id = users.user_id
LIMIT 10;
```
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT user_id, COUNT(*)
FROM cte
GROUP BY user_id
LIMIT 10;
```
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT *
FROM (
SELECT order_id, ROW_NUMBER() OVER (ORDER BY order_id) AS rn
FROM cte
) t
LIMIT 10;
```
```sql
WITH cte AS (
SELECT * FROM orders
)
SELECT *
FROM (
SELECT * FROM cte ORDER BY order_id
) t
LIMIT 10;
```
These operators can change row cardinality or ordering semantics. Unless
other rules have already rewritten the shape into `Limit ->
CTEConsumer` or `Limit -> Project -> CTEConsumer`, this collector skips
them.
(cherry picked from commit d898a1d)
@CalvinKirs
CalvinKirs requested a review from yiguolei as a code ownerJune 2, 2026 07:10
@CalvinKirs

CalvinKirs commented Jun 2, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

1 similar comment
@CalvinKirs

CalvinKirs commented Jun 5, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 77.78% (35/45) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs

CalvinKirs commented Jun 30, 2026

Copy link
Copy Markdown
MemberAuthor

run cloud_p0

@CalvinKirs

CalvinKirs commented Jun 30, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 80.00% (36/45) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 63.64% (35/55) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs

CalvinKirs commented Aug 17, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 80.00% (36/45) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs

CalvinKirs commented Aug 21, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 71.11% (32/45) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs
[fix](nereids) clear consumerIdToLimitRows between CTE rewrite passes
4a66d08
On branch-4.1 ClearContextStatus clears CTE maps one by one instead of
calling clearCteEnvironment() like master, so consumerIdToLimitRows
survived into the second RewriteCteChildren pass. The producer then got
a second ORIGIN limit pushed on top of the already split GLOBAL/LOCAL
limits, and the after-sub-path-push-down job list has no SplitLimit to
split it, leaving PhysicalLimit[ORIGIN] in the final plan.
Fixes limit_push_down.groovy and order_push_down.groovy shape mismatch
(expected PhysicalLimit[GLOBAL], got PhysicalLimit[ORIGIN]).
@CalvinKirs

CalvinKirs commented Aug 24, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 7.75% (33/426) 🎉
Increment coverage report
Complete coverage report

@github-actions

github-actionsBot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 25, 2026
@github-actions

github-actionsBot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@yiguolei
yiguolei merged commit 3bfc343 into branch-4.1Aug 25, 2026
32 of 36 checks passed
@morrySnow
morrySnow deleted the branch-4.1-63675 branch August 31, 2026 07:11
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@CalvinKirs@hello-stephen@yiguolei