[SPARK-38918][SQL][3.2] Nested column pruning should filter out attributes that do not belong to the current relation by allisonwang-db · Pull Request #36386 · apache/spark

allisonwang-db · 2022-04-27T22:48:18Z

What changes were proposed in this pull request?

Backport #36216 to branch-3.2

Why are the changes needed?

To fix a bug in SchemaPruning.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test

viirya · 2022-05-01T19:34:56Z

@allisonwang-db Can you fix the test failure?

allisonwang-db · 2022-05-03T18:12:43Z

@viirya I tried a few times but can't reproduce this TPCDSV1_4_PlanStabilitySuite test failure locally...

viirya · 2022-05-03T18:15:26Z

@allisonwang-db Maybe you can just re-trigger the CI?

viirya · 2022-05-07T05:48:13Z

Hmm, from the logs, seems there are unmatched plans:

2022-05-04T01:30:18.1915772Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m- check simplified (tpcds-v1.4/q5) *** FAILED *** (361 milliseconds)ESC[0mESC[0m
2022-05-04T01:30:18.1916346Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m Plans did not match:ESC[0mESC[0m
2022-05-04T01:30:18.1917069Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/simplified.txtESC[0mESC[0m
2022-05-04T01:30:18.1918103Z ESC[0m[ESC[0mESC[0minfoESC[0m] ESC[0mESC[0mESC[31m last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/explain.txtESC[0mESC[0m

viirya · 2022-05-07T06:24:58Z

@allisonwang-db I can reproduce it locally by running build/sbt "sql/testOnly *PlanStabilitySuite -- -z (tpcds-v1.4/q5)".

viirya · 2022-05-09T17:59:41Z

Seems cosmetic change only in the explain string?

Yes and there is no plan change

viirya

lgtm if CI can pass.

viirya · 2022-05-09T20:53:52Z

CI still doesn't pass:

check simplified (tpcds-v1.4/q4) *** FAILED ***

allisonwang-db · 2022-05-09T22:06:32Z

Hmm let me try again

allisonwang-db · 2022-05-10T16:36:01Z

This test failure is strange. When I ran it individually (for q4 and q5), it failed but when I ran the entire suite together build/sbt "sql/testOnly *PlanStabilitySuite" the individual test case can pass.

dongjoon-hyun

Hi, @allisonwang-db . Could you rebase this PR once more, please?

allisonwang-db · 2022-05-24T02:31:44Z

@dongjoon-hyun Looks like it still has the same issue:

# Test passed
build/sbt "sql/testOnly *PlanStabilitySuite"# Test failed for q4 and q5
build/sbt "sql/testOnly *PlanStabilitySuite -- -z (tpcds-v1.4/q5)"

dongjoon-hyun · 2022-05-24T05:31:46Z

Thank you for rechecking.

olaky · 2022-06-07T12:36:10Z

I am facing the same issues here: #36753

olaky · 2022-06-07T12:43:08Z

So the only change in the plan I can see that makes the test fail is that the last plan node has a source filename in it now, for example

Scan parquet default.web_site [web_site_sk,web_site_id] (PlanStabilitySuite.scala:156)

viirya · 2022-06-10T07:49:36Z

Can #36828 help on stablizing PlanStabilitySuite?

dongjoon-hyun · 2022-06-10T08:32:33Z

+1 for @viirya 's comment. @cloud-fan found the root cause and is fixing now on that PR.

dongjoon-hyun · 2022-06-10T08:32:53Z

We can revisit this PR after merging that PR.

viirya · 2022-06-10T16:39:42Z

@allisonwang-db #36828 is merged, can you rebase to trigger CI? Thanks.

… that do not belong to the current relation This PR updates `ProjectionOverSchema` to use the outputs of the data source relation to filter the attributes in the nested schema pruning. This is needed because the attributes in the schema do not necessarily belong to the current data source relation. For example, if a filter contains a correlated subquery, then the subquery's children can contain attributes from both the inner query and the outer query. Since the `RewriteSubquery` batch happens after early scan pushdown rules, nested schema pruning can wrongly use the inner query's attributes to prune the outer query data schema, thus causing wrong results and unexpected exceptions. To fix a bug in `SchemaPruning`. No Unit test Closesapache#36216 from allisonwang-db/spark-38918-nested-column-pruning. Authored-by: allisonwang-db <allison.wang@databricks.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit 150434b) Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit 793ba60) Signed-off-by: allisonwang-db <allison.wang@databricks.com>

dongjoon-hyun

+1, LGTM.

viirya

Pending CI

…butes that do not belong to the current relation ### What changes were proposed in this pull request? Backport #36216 to branch-3.2 ### Why are the changes needed? To fix a bug in `SchemaPruning`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closes#36386 from allisonwang-db/spark-38918-branch-3.2. Authored-by: allisonwang-db <allison.wang@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2022-06-10T22:50:06Z

Merged to branch-3.2. Thank you so much, @allisonwang-db , @viirya .

…butes that do not belong to the current relation ### What changes were proposed in this pull request? Backport apache#36216 to branch-3.2 ### Why are the changes needed? To fix a bug in `SchemaPruning`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closesapache#36386 from allisonwang-db/spark-38918-branch-3.2. Authored-by: allisonwang-db <allison.wang@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actionsBot added the SQL label Apr 27, 2022

viirya reviewed May 9, 2022
View reviewed changes

viirya approved these changes May 9, 2022
View reviewed changes

dongjoon-hyun reviewed May 22, 2022
View reviewed changes

allisonwang-db force-pushed the spark-38918-branch-3.2 branch from 15c9ccc to 9652212Compare May 23, 2022 18:04

olaky mentioned this pull request Jun 7, 2022
[SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries #36753
Closed

allisonwang-db added 3 commits June 10, 2022 13:27

update explain.txt
ff83285

update golden files
83dba72

allisonwang-db force-pushed the spark-38918-branch-3.2 branch from 9652212 to 83dba72Compare June 10, 2022 20:53

allisonwang-db mentioned this pull request Jun 10, 2022
[3.2][SPARK-39437][SQL][TEST] Normalize plan id separately in PlanStabilitySuite #36828
Closed

dongjoon-hyun approved these changes Jun 10, 2022
View reviewed changes

viirya approved these changes Jun 10, 2022
View reviewed changes

dongjoon-hyun closed this Jun 10, 2022

Uh oh!

Conversation

allisonwang-db commented Apr 27, 2022• edited by viirya Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya commented May 1, 2022

Uh oh!

allisonwang-db commented May 3, 2022

Uh oh!

viirya commented May 3, 2022

Uh oh!

viirya commented May 7, 2022

Uh oh!

viirya commented May 7, 2022

Uh oh!

viiryaMay 9, 2022

Choose a reason for hiding this comment

Uh oh!

allisonwang-dbMay 9, 2022

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

viirya commented May 9, 2022

Uh oh!

allisonwang-db commented May 9, 2022

Uh oh!

allisonwang-db commented May 10, 2022

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

allisonwang-db commented May 24, 2022

Uh oh!

dongjoon-hyun commented May 24, 2022

Uh oh!

olaky commented Jun 7, 2022

Uh oh!

olaky commented Jun 7, 2022

Uh oh!

viirya commented Jun 10, 2022

Uh oh!

dongjoon-hyun commented Jun 10, 2022

Uh oh!

dongjoon-hyun commented Jun 10, 2022

Uh oh!

viirya commented Jun 10, 2022

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

allisonwang-db commented Apr 27, 2022•
edited by viirya
Loading

dongjoon-hyun commented Jun 10, 2022•
edited
Loading