[SPARK-46741][SQL] Cache Table with CTE won't work by AngersZhuuuu · Pull Request #44767 · apache/spark

AngersZhuuuu · 2024-01-17T09:04:44Z

What changes were proposed in this pull request?

Cache Table with CTE won't work, there are two reasons

In the current code CTE in CacheTableAsSelect will be inlined
CTERelation Ref and Def didn't handle the CTEId doCanonicalize issue

Cause the current case can't be matched.

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

Yea, Cache table with CTE can work after this pr

How was this patch tested?

Added UT

Was this patch authored or co-authored using generative AI tooling?

No

AngersZhuuuu · 2024-01-17T09:04:52Z

ping @cloud-fan

AngersZhuuuu · 2024-01-18T02:34:33Z

How about current? @cloud-fan

cloud-fan · 2024-01-19T03:31:28Z

+ _.containsAnyPattern(CTE, PLAN_EXPRESSION)) {
+ case ref: CTERelationRef =>
+ ref.copy(cteId = defIndex(ref.cteId).toLong)
+


can we make sure there is no nested WithCTE?

can we make sure there is no nested WithCTE?
Form the CTESubtitution's code, and it's pr, there won't have nested WithCTE. #37751 cc @maryannxue

shall we add assert to guarantee this assumption?

@cloud-fan Found nested WithCTE case, added to cte.sql and here change the method to support nested CTE, pls take a look again.

AngersZhuuuu · 2024-05-14T02:11:11Z

ping @cloud-fan @yaooqinn

This reverts commit 054231f.

AngersZhuuuu · 2024-07-02T09:25:39Z

ping @cloud-fan @HyukjinKwon @dongjoon-hyun @viirya

HyukjinKwon · 2024-07-03T00:41:38Z

what was behaviour before? Would be great to show the result before/after

AngersZhuuuu · 2024-07-03T02:03:14Z

what was behaviour before? Would be great to show the result before/after

For the query in cache.sql

EXPLAIN EXTENDED SELECT * FROM cache_nested_cte_table

before this pr, cached table cache_nested_cte_table won't match, will execute again, after this pr, it can match the InMemoryRelation

github-actions · 2024-10-12T00:23:11Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

### What changes were proposed in this pull request? Reopen #44767 Cache Table with CTE won't work, there are two reasons 1. In the current code CTE in CacheTableAsSelect will be inlined 2. CTERelation Ref and Def didn't handle the CTEId doCanonicalize issue Cause the current case can't be matched. ### Why are the changes needed? Fix Bug ### Does this PR introduce _any_ user-facing change? Yea, Cache table with CTE can work after this pr For added `cache.sql` final query `EXPLAIN EXTENDED SELECT * FROM cache_nested_cte_table;` Before this pr, the plan as below, cache won't work. <img width="1067" height="584" alt="截屏2025-12-05 11 22 05" src="https://github.com/user-attachments/assets/045df794-38e2-47d9-848e-cfc3c7525671" /> After this pr <img width="1279" height="824" alt="截屏2025-12-05 11 32 38" src="https://github.com/user-attachments/assets/86f5ab33-67c6-44d0-b5d8-4bec51a2d5b7" /> ### How was this patch tested? Added UT ### Was this patch authored or co-authored using generative AI tooling? No Closes#53333 from AngersZhuuuu/SPARK-46741. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

### What changes were proposed in this pull request? Reopen #44767 Cache Table with CTE won't work, there are two reasons 1. In the current code CTE in CacheTableAsSelect will be inlined 2. CTERelation Ref and Def didn't handle the CTEId doCanonicalize issue Cause the current case can't be matched. ### Why are the changes needed? Fix Bug ### Does this PR introduce _any_ user-facing change? Yea, Cache table with CTE can work after this pr For added `cache.sql` final query `EXPLAIN EXTENDED SELECT * FROM cache_nested_cte_table;` Before this pr, the plan as below, cache won't work. <img width="1067" height="584" alt="截屏2025-12-05 11 22 05" src="https://github.com/user-attachments/assets/045df794-38e2-47d9-848e-cfc3c7525671" /> After this pr <img width="1279" height="824" alt="截屏2025-12-05 11 32 38" src="https://github.com/user-attachments/assets/86f5ab33-67c6-44d0-b5d8-4bec51a2d5b7" /> ### How was this patch tested? Added UT ### Was this patch authored or co-authored using generative AI tooling? No Closes#53333 from AngersZhuuuu/SPARK-46741. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 8f69679) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

[SPARK-46741][SQL] Cache Table with CET won't work
c151b60

github-actionsBot added the SQL label Jan 17, 2024

AngersZhuuuu changed the title ~~[SPARK-46741][SQL] Cache Table with CET won't work~~[SPARK-46741][SQL] Cache Table with CTE won't workJan 17, 2024

Update basicLogicalOperators.scala
b9711ab

cloud-fan reviewed Jan 19, 2024
View reviewed changes

cloud-fan approved these changes Jan 19, 2024
View reviewed changes

AngersZhuuuu added 2 commits January 19, 2024 15:39

Update basicLogicalOperators.scala
38c68d8

update
d587e6d

cloud-fan reviewed Jan 23, 2024
View reviewed changes

Comment threadsql/core/src/test/resources/sql-tests/inputs/cache.sql

AngersZhuuuu added 6 commits February 2, 2024 16:01

Merge branch 'master' into SPARK-46741
8b2a25d

Follow comment to add normalize rule
bb15d32

Merge branch 'master' into SPARK-46741
fa17613

Merge branch 'master' into SPARK-46741
a6e2657

Update QueryExecution.scala
c9017ef

Update cache.sql.out
4aa6aee

AngersZhuuuu added 4 commits May 14, 2024 14:24

Merge branch 'master' into SPARK-46741
99bf379

follow comment
054231f

Update cache.sql.out
92a213d

Revert "follow comment"
123a986
This reverts commit 054231f.

github-actionsBot added the Stale label Oct 12, 2024

github-actionsBot closed this Oct 13, 2024

AngersZhuuuu mentioned this pull request Dec 5, 2025
[SPARK-46741][SQL] Cache Table with CTE won't work #53333
Closed

Navigation Menu

Uh oh!

[SPARK-46741][SQL] Cache Table with CTE won't work - #44767

[SPARK-46741][SQL] Cache Table with CTE won't work#44767
AngersZhuuuu wants to merge 14 commits into
apache:masterfrom
AngersZhuuuu:SPARK-46741

AngersZhuuuu commented Jan 17, 2024

Uh oh!

AngersZhuuuu commented Jan 17, 2024

Uh oh!

AngersZhuuuu commented Jan 18, 2024

Uh oh!

cloud-fanJan 19, 2024

Uh oh!

AngersZhuuuuJan 19, 2024

Uh oh!

cloud-fanJan 19, 2024•
edited
Loading

Uh oh!

AngersZhuuuuJan 19, 2024

Uh oh!

AngersZhuuuuJan 23, 2024

Uh oh!

Uh oh!

AngersZhuuuu commented May 14, 2024

Uh oh!

AngersZhuuuu commented Jul 2, 2024

Uh oh!

HyukjinKwon commented Jul 3, 2024

Uh oh!

AngersZhuuuu commented Jul 3, 2024

Uh oh!

github-actionsBot commented Oct 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AngersZhuuuu commented Jan 17, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

AngersZhuuuu commented Jan 17, 2024

Uh oh!

AngersZhuuuu commented Jan 18, 2024

Uh oh!

cloud-fanJan 19, 2024

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuuJan 19, 2024

Choose a reason for hiding this comment

Uh oh!

cloud-fanJan 19, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuuJan 19, 2024

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuuJan 23, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AngersZhuuuu commented May 14, 2024

Uh oh!

AngersZhuuuu commented Jul 2, 2024

Uh oh!

HyukjinKwon commented Jul 3, 2024

Uh oh!

AngersZhuuuu commented Jul 3, 2024

Uh oh!

github-actionsBot commented Oct 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cloud-fanJan 19, 2024•
edited
Loading