[SPARK-26424][SQL] Use java.time API in date/timestamp expressions by MaxGekk · Pull Request #23358 · apache/spark

MaxGekk · 2018-12-20T22:22:55Z

What changes were proposed in this pull request?

In the PR, I propose to switch the DateFormatClass, ToUnixTimestamp, FromUnixTime, UnixTime on java.time API for parsing/formatting dates and timestamps. The API has been already implemented by the Timestamp/DateFormatter classes. One of benefit is those classes support parsing timestamps with microsecond precision. Old behaviour can be switched on via SQL config: spark.sql.legacy.timeParser.enabled (false by default).

How was this patch tested?

It was tested by existing test suites - DateFunctionsSuite, DateExpressionsSuite, JsonSuite, CsvSuite, SQLQueryTestSuite as well as PySpark tests.

SparkQA · 2018-12-21T00:07:20Z

Test build #100346 has finished for PR 23358 at commit 964780f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit 9cd8f33.

SparkQA · 2018-12-21T21:38:05Z

Test build #100363 has finished for PR 23358 at commit 9cd8f33.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-12-21T21:40:43Z

Test build #100369 has finished for PR 23358 at commit 30d9226.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-12-21T22:19:26Z

+ .parseCaseInsensitive()
 .appendPattern(pattern)
- .parseDefaulting(ChronoField.YEAR_OF_ERA, 1970)
+ .parseDefaulting(ChronoField.ERA, 1)


Era is required in STRICT mode

is 1 a reasonable default value for ERA?

I think so. This is our current era: https://docs.oracle.com/javase/8/docs/api/java/time/temporal/ChronoField.html#ERA : "The value of the era that was active on 1970-01-01 (ISO) must be assigned the value 1."

MaxGekk · 2018-12-21T22:21:11Z

 new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
 .appendPattern(pattern)
- .parseDefaulting(ChronoField.YEAR_OF_ERA, 1970)


Year must always present in timestamps/dates. Probability of an user is satisfied to default value 1970 is pretty low. Don't think if the user wants to parse let's say 14 Nov, he/she means 14 Nov 1970. I would guess current year but this approach is error prone.

MaxGekk · 2018-12-21T22:25:42Z

+ .parseDefaulting(ChronoField.ERA, 1)
 .parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
 .parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
- .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)


Hours must always present in the time part. The default value causes conflict if the timestamp pattern has a (AM or PM). If there are no hours, we set the time part to zero later.

MaxGekk · 2018-12-21T22:30:47Z


 protected def toInstantWithZoneId(temporalAccessor: TemporalAccessor, zoneId: ZoneId): Instant = {
- val localDateTime = LocalDateTime.from(temporalAccessor)
+ val localTime = if (temporalAccessor.query(TemporalQueries.localTime) == null) {


If parsed timestamp does not have the time part at all, set all (hours, minutes, seconds and etc.) to zeros.

MaxGekk · 2018-12-21T22:32:47Z

+ */
+ @throws(classOf[ParseException])
+ @throws(classOf[DateTimeParseException])
+ @throws(classOf[DateTimeException])


This annotations are required in whole stage codegen otherwise I got an error about some exception catches are not reachable/used.

MaxGekk · 2018-12-21T22:33:57Z

 locale: Locale) extends TimestampFormatter with DateTimeFormatterHelper {
- private val formatter = buildFormatter(pattern, locale)
+ @transient
+ private lazy val formatter = buildFormatter(pattern, locale)


The Iso8601TimestampFormatter class became serializable but the build is still not.

MaxGekk · 2018-12-21T22:35:52Z

 import org.apache.spark.sql.internal.SQLConf

-sealed trait TimestampFormatter {
+sealed trait TimestampFormatter extends Serializable {


Making it serializable otherwise I got task not serializable exception from generated code

SparkQA · 2018-12-21T22:42:28Z

Test build #100371 has finished for PR 23358 at commit e4baad6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-12-21T22:43:53Z

jenkins, retest this, please

SparkQA · 2018-12-22T17:21:57Z

Test build #100398 has finished for PR 23358 at commit 2ac29d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-12-24T09:48:28Z

As @hvanhovell mentioned offline, the implementation based on java.time classes changes behavior since it uses IsoChronology which is actually Proleptic Gregorian calendar. It could cause some problems in manipulating old dates. I am going to mention that in the migration guide.

I think we can support other calendars like Julian calendar the future by using libraries like ThreeTen-Extra.

…ime-cast

MaxGekk · 2018-12-24T11:24:25Z

R build failed:

> devtools::install_version('testthat', version = '1.0.2', repos='http://cran.us.r-project.org')
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) : there is no package called 'pkgbuild'

MaxGekk · 2018-12-24T13:13:21Z

jenkins, retest this, please

SparkQA · 2018-12-24T15:08:43Z

Test build #100419 has finished for PR 23358 at commit d9d3616.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-12-24T15:10:58Z

Test build #100424 has finished for PR 23358 at commit d9d3616.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…ime-cast

SparkQA · 2018-12-24T19:30:19Z

Test build #100426 has finished for PR 23358 at commit 0e1afc3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2018-12-25T03:30:48Z

The migration guide update LGTM.

Do we have an example to show the calendar problem? AFAIK SQL standard follows Gregorian calendar, so we shouldn't support Julian calendar.

MaxGekk · 2018-12-25T09:30:01Z

Do we have an example to show the calendar problem?

There is a shift in dates:

and number of days in year is different - 365.2425 days in Gregorian calendar vs 365.25 days in Julian calendar. So, if you have a day arithmetic, you get different results (in years) in the calendars.

cloud-fan · 2018-12-26T12:36:24Z

LGTM

cloud-fan · 2018-12-27T03:10:38Z

thanks, merging to master!

dongjoon-hyun · 2018-12-28T10:30:54Z

Hi, All.

This PR seems to break code generation in Scala-2.11 profile.

I'll make a followup PR to fix this.

## What changes were proposed in this pull request? In the PR, I propose to switch the `DateFormatClass`, `ToUnixTimestamp`, `FromUnixTime`, `UnixTime` on java.time API for parsing/formatting dates and timestamps. The API has been already implemented by the `Timestamp`/`DateFormatter` classes. One of benefit is those classes support parsing timestamps with microsecond precision. Old behaviour can be switched on via SQL config: `spark.sql.legacy.timeParser.enabled` (`false` by default). ## How was this patch tested? It was tested by existing test suites - `DateFunctionsSuite`, `DateExpressionsSuite`, `JsonSuite`, `CsvSuite`, `SQLQueryTestSuite` as well as PySpark tests. Closesapache#23358 from MaxGekk/new-time-cast. Lead-authored-by: Maxim Gekk <maxim.gekk@databricks.com> Co-authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

## What changes were proposed in this pull request? This PR fixes the codegen bug introduced by apache#23358 . - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.11/158/ ``` Line 44, Column 93: A method named "apply" is not declared in any enclosing class nor any supertype, nor through a static import ``` ## How was this patch tested? Manual. `DateExpressionsSuite` should be passed with Scala-2.11. Closesapache#23394 from dongjoon-hyun/SPARK-26424. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

## What changes were proposed in this pull request? In the PR, I propose to switch the `DateFormatClass`, `ToUnixTimestamp`, `FromUnixTime`, `UnixTime` on java.time API for parsing/formatting dates and timestamps. The API has been already implemented by the `Timestamp`/`DateFormatter` classes. One of benefit is those classes support parsing timestamps with microsecond precision. Old behaviour can be switched on via SQL config: `spark.sql.legacy.timeParser.enabled` (`false` by default). ## How was this patch tested? It was tested by existing test suites - `DateFunctionsSuite`, `DateExpressionsSuite`, `JsonSuite`, `CsvSuite`, `SQLQueryTestSuite` as well as PySpark tests. Closesapache#23358 from MaxGekk/new-time-cast. Lead-authored-by: Maxim Gekk <maxim.gekk@databricks.com> Co-authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

## What changes were proposed in this pull request? This PR fixes the codegen bug introduced by apache#23358 . - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.11/158/ ``` Line 44, Column 93: A method named "apply" is not declared in any enclosing class nor any supertype, nor through a static import ``` ## How was this patch tested? Manual. `DateExpressionsSuite` should be passed with Scala-2.11. Closesapache#23394 from dongjoon-hyun/SPARK-26424. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

MaxGekk added 7 commits December 20, 2018 19:08

Porting DateFormatClass on new formatter
5ea78cd

Porting UnixTime on new formatter
162252f

Porting FromUnixTime on new formatter
fd6f7cc

Fix task not serializable
0089351

Catch illigalargexception
5150615

Remove unused method newDateFormat
29e94b4

Set invalid date for Feb
964780f

HyukjinKwon reviewed Dec 21, 2018
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala Outdated

MaxGekk added 6 commits December 21, 2018 18:28

Enable legacy parser for Hive Compatibility suite
9cd8f33

Test for parsing AM/PM
5d52c0e

Handling DateTimeException
f444f57

Set default time if it is not parsed
6d90b30

Revert "Enable legacy parser for Hive Compatibility suite"
30d9226
This reverts commit 9cd8f33.

Enable STRICT mode
e4baad6

Set default values for month, day, minute and second.
c3fe2a7

MaxGekk commented Dec 21, 2018
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala

MaxGekk commented Dec 21, 2018
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala

MaxGekk commented Dec 21, 2018
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

cloud-fan reviewed Dec 24, 2018
View reviewed changes

Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/util/DateFormatterSuite.scala Outdated

cloud-fan reviewed Dec 24, 2018
View reviewed changes

Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala Outdated

cloud-fan approved these changes Dec 24, 2018
View reviewed changes

MaxGekk added 3 commits December 24, 2018 11:32

Addressing Wenchen's comments
6bc9f54

Updating the migration guide regarding calendars
fb21d93

Merge branch 'new-time-cast' of github.com:MaxGekk/spark-1 into new-t…
d9d3616
…ime-cast

MaxGekk added 2 commits December 24, 2018 16:29

Merge remote-tracking branch 'origin/master' into new-time-cast
d425570

Merge branch 'new-time-cast' of github.com:MaxGekk/spark-1 into new-t…
0e1afc3
…ime-cast

HyukjinKwon approved these changes Dec 27, 2018
View reviewed changes

asfgit closed this in 7c7fccfDec 27, 2018

dongjoon-hyun mentioned this pull request Dec 28, 2018
[SPARK-26424][SQL][FOLLOWUP] Fix DateFormatClass/UnixTime codegen #23394
Closed

MaxGekk mentioned this pull request Dec 29, 2018
[SPARK-26002][SQL] Fix day of year calculation for Julian calendar days #23000
Closed

MaxGekk deleted the new-time-cast branch August 17, 2019 13:35

Uh oh!

Conversation

MaxGekk commented Dec 20, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Dec 21, 2018

Uh oh!

Uh oh!

SparkQA commented Dec 21, 2018

Uh oh!

SparkQA commented Dec 21, 2018

Uh oh!

MaxGekkDec 21, 2018

Choose a reason for hiding this comment

Uh oh!

cloud-fanDec 26, 2018

Choose a reason for hiding this comment

Uh oh!

MaxGekkDec 26, 2018

Choose a reason for hiding this comment

Uh oh!

MaxGekkDec 21, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekkDec 21, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MaxGekkDec 21, 2018

Choose a reason for hiding this comment

Uh oh!

MaxGekkDec 21, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekkDec 21, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MaxGekkDec 21, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Dec 21, 2018

Uh oh!

MaxGekk commented Dec 21, 2018

Uh oh!

SparkQA commented Dec 22, 2018

Uh oh!

Uh oh!

Uh oh!

MaxGekk commented Dec 24, 2018

Uh oh!

MaxGekk commented Dec 24, 2018

Uh oh!

MaxGekk commented Dec 24, 2018

Uh oh!

SparkQA commented Dec 24, 2018

Uh oh!

SparkQA commented Dec 24, 2018

Uh oh!

SparkQA commented Dec 24, 2018

Uh oh!

cloud-fan commented Dec 25, 2018

Uh oh!

MaxGekk commented Dec 25, 2018

Uh oh!

cloud-fan commented Dec 26, 2018

Uh oh!

cloud-fan commented Dec 27, 2018

Uh oh!

dongjoon-hyun commented Dec 28, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

MaxGekkDec 21, 2018•
edited
Loading

MaxGekkDec 21, 2018•
edited
Loading

MaxGekkDec 21, 2018•
edited
Loading