[SPARK-28141][SQL] Support special date values by MaxGekk · Pull Request #25708 · apache/spark

MaxGekk · 2019-09-06T08:46:43Z

What changes were proposed in this pull request?

Supported special string values for DATE type. They are simply notational shorthands that will be converted to ordinary date values when read. The following string values are supported:

epoch [zoneId] - 1970-01-01
today [zoneId] - the current date in the time zone specified by spark.sql.session.timeZone.
yesterday [zoneId] - the current date -1
tomorrow [zoneId] - the current date + 1
now - the date of running the current query. It has the same notion as today.

For example:

spark-sql>SELECTdate'tomorrow'-date'yesterday';
2

Why are the changes needed?

To maintain feature parity with PostgreSQL, see 8.5.1.4. Special Values

Does this PR introduce any user-facing change?

Previously, the parser fails on the special values with the error:

spark-sql>selectdate'today';
Error in query: Cannot parse the DATE value: today(line1, pos 7)

After the changes, the special values are converted to appropriate dates:

spark-sql>selectdate'today';
2019-09-06

How was this patch tested?

Added tests to DateFormatterSuite to check parsing special values from regular strings.
Tests in DateTimeUtilsSuite check parsing those values from UTF8String
Uncommented tests in date.sql

SparkQA · 2019-09-06T11:29:43Z

Test build #110230 has finished for PR 25708 at commit 775c7ec.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-06T11:36:44Z

+
+ assert(toDate("epoch", zoneId).get === 0)
+ val today = localDateToDays(LocalDate.now(zoneId))
+ assert(toDate("yesterday", zoneId).get === today - 1)


Here, there is the risk that today contains previous day if we were unlucky and the test was executed at the end of the day. Maybe, need to introduce some tolerance.

SparkQA · 2019-09-06T14:59:34Z

Test build #110236 has finished for PR 25708 at commit be787f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…ecial-values # Conflicts: # sql/core/src/test/resources/sql-tests/results/pgSQL/date.sql.out

SparkQA · 2019-09-06T20:25:58Z

Test build #110255 has finished for PR 25708 at commit 4fb8834.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-06T23:16:53Z

Test build #110258 has finished for PR 25708 at commit 6da7905.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-07T17:19:28Z

@dongjoon-hyun @maropu @cloud-fan Please, review the PR.

SparkQA · 2019-09-07T21:03:28Z

Test build #110285 has finished for PR 25708 at commit d57e11e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-09-08T00:04:11Z

The other DBMS-like systems support these syntaxes? If not, we should turn on/off these features by the compatibility mode flag https://issues.apache.org/jira/browse/SPARK-28934?

maropu · 2019-09-08T00:50:00Z

How about the timestamp case?
You dropped inf/-inf in the supported?

maropu · 2019-09-08T01:15:56Z

+ case _ =>
+ (c, evPrim, evNull) => code"$evNull = true;"
+ }
 }


Just put the gen'd code for other reviewers;
/* 051 */ boolean project_isNull_0 = columnartorow_isNull_0; /* 052 */ int project_value_0 = -1; /* 053 */ if (!columnartorow_isNull_0) { /* 054 */ scala.Option<Integer> project_intOpt_0 = /* 055 */ org.apache.spark.sql.catalyst.util.DateTimeUtils.stringToDate(columnartorow_value_0, ((java.time.ZoneId) references[2] /* zoneId */)); /* 056 */ if (project_intOpt_0.isDefined()) { /* 057 */ project_value_0 = ((Integer) project_intOpt_0.get()).intValue(); /* 058 */ } else { /* 059 */ project_isNull_0 = true; /* 060 */ } /* 061 */ }

maropu · 2019-09-08T01:22:26Z

+ val zid = getZoneId()
+ (c, evPrim, evNull) =>
+ code"""
+ scala.Option<Integer> $intOpt =


btw, (this is not related to this pr though), scala.Option<Integer> is ok for java compilers/jvm? @rednaxelafx I remember that jdk compilers cannot compile this statement: #21770 (comment) because it seems the compilers erase the returned type from scala.Option<Integer> to scala.Option<Object>. I'm not sure why janino accepts this though....

SparkQA · 2019-09-10T09:13:41Z

Test build #110409 has finished for PR 25708 at commit 106524b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-09-10T11:39:04Z

retest this please

SparkQA · 2019-09-10T15:19:25Z

Test build #110417 has finished for PR 25708 at commit 106524b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-12T07:15:03Z

@maropu @dongjoon-hyun Here is the same question as in #25716 (comment)

…ecial-values # Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala # sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala # sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala

SparkQA · 2019-09-18T20:43:57Z

Test build #110922 has finished for PR 25708 at commit ff92531.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-09-19T05:44:23Z

I have rebased this on the master with merged #25716 . @HyukjinKwon Could you take a look at this, please.

MaxGekk · 2019-09-20T18:25:10Z

@dongjoon-hyun @maropu @HyukjinKwon Could take a look at this PR which is similar to already merged changes for the timestamp type.

maropu

Looks nice cc: @HyukjinKwon @dongjoon-hyun

HyukjinKwon · 2019-09-22T08:29:50Z

I am merging this just to unblock and easier to work further but please make sure to have a flag. Seems like that's pointed out by multiple committers such as @maropu and @cloud-fan.

HyukjinKwon · 2019-09-22T08:30:08Z

Otherwise, I will revert this and #25716

Merged to master.

maropu · 2019-09-22T09:44:01Z

We already have a jira ticket for linking this feature to the flag?

MaxGekk · 2019-09-22T17:07:28Z

This PR #25834 hides the feature under the SQL config spark.sql.dialect = "PostgreSQL"

maropu · 2019-09-22T23:04:03Z

Oh, I see. Thanks!

… SQL migration guide ### What changes were proposed in this pull request? Updated the SQL migration guide regarding to recently supported special date and timestamp values, see #25716 and #25708. Closes#25834 ### Why are the changes needed? To let users know about new feature in Spark 3.0. ### Does this PR introduce any user-facing change? No Closes#25948 from MaxGekk/special-values-migration-guide. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…only ### What changes were proposed in this pull request? In the PR, I propose to support special datetime values introduced by #25708 and by #25716 only in typed literals, and don't recognize them in parsing strings to dates/timestamps. The following string values are supported only in typed timestamp literals: - `epoch [zoneId]` - `1970-01-01 00:00:00+00 (Unix system time zero)` - `today [zoneId]` - midnight today. - `yesterday [zoneId]` - midnight yesterday - `tomorrow [zoneId]` - midnight tomorrow - `now` - current query start time. For example: ```sql spark-sql> SELECT timestamp 'tomorrow'; 2019-09-07 00:00:00 ``` Similarly, the following special date values are supported only in typed date literals: - `epoch [zoneId]` - `1970-01-01` - `today [zoneId]` - the current date in the time zone specified by `spark.sql.session.timeZone`. - `yesterday [zoneId]` - the current date -1 - `tomorrow [zoneId]` - the current date + 1 - `now` - the date of running the current query. It has the same notion as `today`. For example: ```sql spark-sql> SELECT date 'tomorrow' - date 'yesterday'; 2 ``` ### Why are the changes needed? In the current implementation, Spark supports the special date/timestamp value in any input strings casted to dates/timestamps that leads to the following problems: - If executors have different system time, the result is inconsistent, and random. Column values depend on where the conversions were performed. - The special values play the role of distributed non-deterministic functions though users might think of the values as constants. ### Does this PR introduce _any_ user-facing change? Yes but the probability should be small. ### How was this patch tested? By running existing test suites: ``` $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z interval.sql" $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z date.sql" $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z timestamp.sql" $ build/sbt "test:testOnly *DateTimeUtilsSuite" ``` Closes#32714 from MaxGekk/remove-datetime-special-values. Lead-authored-by: Max Gekk <max.gekk@gmail.com> Co-authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

MaxGekk added 7 commits September 5, 2019 22:21

Support special date values
c5034a8

Use special values in stringToDate
695a6c6

Tests for stringToDate
ebc5b02

Use map with UTF8String keys
2663542

Bug fix: pass zoneId in generated code
edda1f2

Uncomment tests in date.sql
8d023ec

Change ticket for the special values: infinity and -infinity
775c7ec

MaxGekk commented Sep 6, 2019
View reviewed changes

Bug fix: pass zoneId to Cast
be787f4

MaxGekk mentioned this pull request Sep 6, 2019
[WIP][SQL] Resolve the time zone before inline tables #25711
Closed

MaxGekk added 7 commits September 6, 2019 21:57

Bug fix: use passed zoneId parameter
f356b72

Remove an unused import
864e456

Use outstandingZoneIds in the test
e237b27

Merge remote-tracking branch 'remotes/origin/master' into datetime-sp…
0055a8a
…ecial-values # Conflicts: # sql/core/src/test/resources/sql-tests/results/pgSQL/date.sql.out

Re-get date.sql.out
4fb8834

Create zoneId when it is needed
9a2349b

Bug fix: require time zone in the string to date conversion
6da7905

dongjoon-hyun added the SQL label Sep 6, 2019

Use regex
d57e11e

MaxGekk commented Sep 7, 2019
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala

maropu reviewed Sep 8, 2019
View reviewed changes

MaxGekk added 2 commits September 18, 2019 20:35

Remove an unused import
ff92531

MaxGekk mentioned this pull request Sep 18, 2019
[WIP][SPARK-29155][SQL] Support special date/timestamp values in the PostgreSQL dialect only #25834
Closed

maropu approved these changes Sep 20, 2019
View reviewed changes

HyukjinKwon closed this in 051e691Sep 22, 2019

MaxGekk mentioned this pull request Sep 27, 2019
[SPARK-29275][SQL][DOC] Describe special date/timestamp values in the SQL migration guide #25948
Closed

MaxGekk deleted the datetime-special-values branch October 5, 2019 19:18

MaxGekk mentioned this pull request May 31, 2021
[SPARK-35581][SQL] Support special datetime values in typed literals only #32714
Closed

Uh oh!

Conversation

MaxGekk commented Sep 6, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Sep 6, 2019

Uh oh!

MaxGekkSep 6, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 6, 2019

Uh oh!

SparkQA commented Sep 6, 2019

Uh oh!

SparkQA commented Sep 6, 2019

Uh oh!

MaxGekk commented Sep 7, 2019

Uh oh!

Uh oh!

SparkQA commented Sep 7, 2019

Uh oh!

maropu commented Sep 8, 2019

Uh oh!

maropu commented Sep 8, 2019

Uh oh!

maropuSep 8, 2019

Choose a reason for hiding this comment

Uh oh!

maropuSep 8, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 10, 2019

Uh oh!

maropu commented Sep 10, 2019

Uh oh!

SparkQA commented Sep 10, 2019

Uh oh!

MaxGekk commented Sep 12, 2019

Uh oh!

SparkQA commented Sep 18, 2019

Uh oh!

MaxGekk commented Sep 19, 2019

Uh oh!

MaxGekk commented Sep 20, 2019

Uh oh!

maropu left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Sep 22, 2019

Uh oh!

HyukjinKwon commented Sep 22, 2019

Uh oh!

maropu commented Sep 22, 2019

Uh oh!

MaxGekk commented Sep 22, 2019

Uh oh!

maropu commented Sep 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

MaxGekk commented Sep 6, 2019•
edited
Loading

MaxGekkSep 6, 2019•
edited
Loading