[SPARK-36567][SQL] Support foldable special datetime strings by `CAST` by MaxGekk · Pull Request #33816 · apache/spark

MaxGekk · 2021-08-23T20:03:44Z

What changes were proposed in this pull request?

In the PR, I propose to add new correctness rule SpecialDatetimeValues to the final analysis phase. It replaces casts of strings to date/timestamp_ltz/timestamp_ntz by literals of such types if the strings contain special datetime values like today, yesterday and tomorrow, and the input strings are foldable.

Why are the changes needed?

To avoid a breaking change.
To improve user experience with Spark SQL. After the PR [SPARK-35581][SQL] Support special datetime values in typed literals only #32714, users have to use typed literals instead of implicit casts. For instance,
at Spark 3.1:

select ts_col >'now';

but the query fails at the moment, and users have to use typed timestamp literal:

select ts_col >timestamp'now';

Does this PR introduce any user-facing change?

No. Previous release 3.1 has supported the feature already till it was removed by #32714.

How was this patch tested?

Manually tested via the sql command line:

spark-sql>select cast('today'asdate);
2021-08-24
spark-sql>selecttimestamp('today');
2021-08-2400:00:00
spark-sql>selecttimestamp'tomorrow'>'today';
true

By running new test suite:

$ build/sbt "sql/testOnly org.apache.spark.sql.catalyst.optimizer.SpecialDatetimeValuesSuite"

SparkQA · 2021-08-23T20:32:43Z

Test build #142708 has finished for PR 33816 at commit 16371f7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-23T20:51:10Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47209/

SparkQA · 2021-08-23T21:26:20Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47209/

SparkQA · 2021-08-23T22:28:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47210/

SparkQA · 2021-08-23T23:21:13Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47210/

SparkQA · 2021-08-24T02:12:26Z

Test build #142709 has finished for PR 33816 at commit 59933d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-24T18:36:38Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47241/

SparkQA · 2021-08-24T19:30:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47241/

MaxGekk · 2021-08-24T19:48:53Z

@cloud-fan @gengliangwang @HyukjinKwon Could you review this PR, please.

SparkQA · 2021-08-24T22:30:16Z

Test build #142741 has finished for PR 33816 at commit d7a1787.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-08-25T06:08:58Z

thanks, merging to master/3.2!

### What changes were proposed in this pull request? In the PR, I propose to add new correctness rule `SpecialDatetimeValues` to the final analysis phase. It replaces casts of strings to date/timestamp_ltz/timestamp_ntz by literals of such types if the strings contain special datetime values like `today`, `yesterday` and `tomorrow`, and the input strings are foldable. ### Why are the changes needed? 1. To avoid a breaking change. 2. To improve user experience with Spark SQL. After the PR #32714, users have to use typed literals instead of implicit casts. For instance, at Spark 3.1: ```sql select ts_col > 'now'; ``` but the query fails at the moment, and users have to use typed timestamp literal: ```sql select ts_col > timestamp'now'; ``` ### Does this PR introduce _any_ user-facing change? No. Previous release 3.1 has supported the feature already till it was removed by #32714. ### How was this patch tested? 1. Manually tested via the sql command line: ```sql spark-sql> select cast('today' as date); 2021-08-24 spark-sql> select timestamp('today'); 2021-08-24 00:00:00 spark-sql> select timestamp'tomorrow' > 'today'; true ``` 2. By running new test suite: ``` $ build/sbt "sql/testOnly org.apache.spark.sql.catalyst.optimizer.SpecialDatetimeValuesSuite" ``` Closes#33816 from MaxGekk/foldable-datetime-special-values. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit df0ec56) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan · 2021-08-25T06:11:04Z

@MaxGekk I think we should also update the migration guide?

MaxGekk · 2021-08-25T20:31:29Z

Here is the update for the SQL migration guide #33840

…ion guide about foldable special datetime values ### What changes were proposed in this pull request? In the PR, I propose to update an existing item in the SQL migration guide, and mention that Spark 3.2 supports foldable special datetime values as well. <img width="1292" alt="Screenshot 2021-08-25 at 23 29 51" src="https://user-images.githubusercontent.com/1580697/130860184-27f0ba56-6c2d-4a5a-91a8-195f2f8aa5da.png"> ### Why are the changes needed? To inform users about actual Spark SQL behavior introduced by #33816 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By generating docs, and checking results manually. Closes#33840 from MaxGekk/special-datetime-cast-migr-guide. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…ion guide about foldable special datetime values ### What changes were proposed in this pull request? In the PR, I propose to update an existing item in the SQL migration guide, and mention that Spark 3.2 supports foldable special datetime values as well. <img width="1292" alt="Screenshot 2021-08-25 at 23 29 51" src="https://user-images.githubusercontent.com/1580697/130860184-27f0ba56-6c2d-4a5a-91a8-195f2f8aa5da.png"> ### Why are the changes needed? To inform users about actual Spark SQL behavior introduced by #33816 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By generating docs, and checking results manually. Closes#33840 from MaxGekk/special-datetime-cast-migr-guide. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit c4e739f) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

Support foldable special datetimes in CAST
16371f7

github-actionsBot added the SQL label Aug 23, 2021

Fix NPE
59933d2

HyukjinKwon reviewed Aug 24, 2021
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala

MaxGekk added 2 commits August 24, 2021 19:20

Add SpecialDatetimeValuesSuite
9a9bd1a

Add timestamp tests
d7a1787

MaxGekk changed the title ~~[WIP][SPARK-36567][SQL] Support foldable special datetime strings by CAST~~[SPARK-36567][SQL] Support foldable special datetime strings by CASTAug 24, 2021

MaxGekk marked this pull request as ready for review August 24, 2021 19:27

cloud-fan approved these changes Aug 25, 2021
View reviewed changes

cloud-fan closed this in df0ec56Aug 25, 2021

MaxGekk mentioned this pull request Aug 25, 2021
[SPARK-35581][SPARK-36567][SQL][DOCS][FOLLOWUP] Update the SQL migration guide about foldable special datetime values #33840
Closed

Uh oh!

Conversation

MaxGekk commented Aug 23, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Aug 23, 2021

Uh oh!

SparkQA commented Aug 23, 2021

Uh oh!

SparkQA commented Aug 23, 2021

Uh oh!

SparkQA commented Aug 23, 2021

Uh oh!

SparkQA commented Aug 23, 2021

Uh oh!

SparkQA commented Aug 24, 2021

Uh oh!

Uh oh!

SparkQA commented Aug 24, 2021

Uh oh!

SparkQA commented Aug 24, 2021

Uh oh!

MaxGekk commented Aug 24, 2021

Uh oh!

SparkQA commented Aug 24, 2021

Uh oh!

cloud-fan commented Aug 25, 2021

Uh oh!

cloud-fan commented Aug 25, 2021

Uh oh!

MaxGekk commented Aug 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MaxGekk commented Aug 23, 2021•
edited
Loading