Skip to content

[SPARK-48031][SQL][FOLLOW-UP] Use ANSI-enabled cast in view lookup test - #46614

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-48031-followup
Closed

[SPARK-48031][SQL][FOLLOW-UP] Use ANSI-enabled cast in view lookup test#46614
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-48031-followup

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR is a followup of #46267 that uses ANSI-enabled cast in the tests. It intentionally uses ANSI-enabled cast in castColToType when you look up a view.

Why are the changes needed?

In order to fix the scheduled CI build without ANSI:

[info] - look up view relation *** FAILED *** (72 milliseconds)
[info]   == FAIL: Plans do not match ===
[info]    'SubqueryAlias spark_catalog.db3.view1                                                                                                                                                                                                                                                                                                                                                                          'SubqueryAlias spark_catalog.db3.view1
[info]    +- View (`spark_catalog`.`db3`.`view1`, ['col1, 'col2, 'a, 'b])                                                                                                                                                                                                                                                                                                                                                 +- View (`spark_catalog`.`db3`.`view1`, ['col1, 'col2, 'a, 'b])
[info]       +- 'Project [cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, col1, 0, 1) as int) AS col1#0, cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, col2, 0, 1) as string) AS col2#0, cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, a, 0, 1) as int) AS a#0, cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, b, 0, 1) as string) AS b#0]      +- 'Project [cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, col1, 0, 1) as int) AS col1#0, cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, col2, 0, 1) as string) AS col2#0, cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, a, 0, 1) as int) AS a#0, cast(getviewcolumnbynameandordinal(`spark_catalog`.`db3`.`view1`, b, 0, 1) as string) AS b#0]
[info]          +- 'Project [*]                                                                                                                                                                                                                                                                                                                                                                                                 +- 'Project [*]
[info]             +- 'UnresolvedRelation [tbl1], [], false        
[info] - look up view created before Spark 3.0 *** FAILED *** (452 milliseconds)
[info]   == FAIL: Plans do not match ===
[info]    'SubqueryAlias spark_catalog.db3.view2                                                                                                                                                                                                                                                                                                          'SubqueryAlias spark_catalog.db3.view2
[info]    +- View (`db3`.`view2`, ['col1, 'col2, 'a, 'b])                                                                                                                                                                                                                                                                                                 +- View (`db3`.`view2`, ['col1, 'col2, 'a, 'b])
[info]       +- 'Project [cast(getviewcolumnbynameandordinal(`db3`.`view2`, col1, 0, 1) as int) AS col1#0, cast(getviewcolumnbynameandordinal(`db3`.`view2`, col2, 0, 1) as string) AS col2#0, cast(getviewcolumnbynameandordinal(`db3`.`view2`, a, 0, 1) as int) AS a#0, cast(getviewcolumnbynameandordinal(`db3`.`view2`, b, 0, 1) as string) AS b#0]      +- 'Project [cast(getviewcolumnbynameandordinal(`db3`.`view2`, col1, 0, 1) as int) AS col1#0, cast(getviewcolumnbynameandordinal(`db3`.`view2`, col2, 0, 1) as string) AS col2#0, cast(getviewcolumnbynameandordinal(`db3`.`view2`, a, 0, 1) as int) AS a#0, cast(getviewcolumnbynameandordinal(`db3`.`view2`, b, 0, 1) as string) AS b#0]
[info]          +- 'Project [*]                                                                                                                                                                                                                                                                                                                                 +- 'Project [*]
[info]             +- 'UnresolvedRelation [tbl1], [], false                                                                                                                                                                                                                                                                                                        +- 'UnresolvedRelation [tbl1], [], false (PlanTest.scala:179)

Does this PR introduce any user-facing change?

No, the main change has not been released yet.

How was this patch tested?

Manually ran the tests after ANSI disabled.

Was this patch authored or co-authored using generative AI tooling?

No.

@srielau

srielau commented May 16, 2024

Copy link
Copy Markdown
Contributor

@HyukjinKwon Any idea why this did not trip the original PR?

@HyukjinKwon

HyukjinKwon commented May 16, 2024

Copy link
Copy Markdown
Member Author

because we don't run the tests without ANSI in PR builder (because of resource limitation). It only runs in the daily build defined here https://github.com/apache/spark/actions/workflows/build_non_ansi.yml

@HyukjinKwon

Copy link
Copy Markdown
Member Author

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants