[SPARK-33500][SQL] Support field "EPOCH" in datetime function extract/date_part

[gengliangwang]

What changes were proposed in this pull request?

Support filed EPOCH in the function extract and date_part, which returns the number of seconds since 1970-01-01 00:00:00-00
For example:

> SELECT EXTRACT(EPOCH FROM TIMESTAMP'1970-01-02 00:00:00-08');
115200.0
> SELECT DATE_PART('EPOCH', TIMESTAMP'1970-01-01 00:00:01.234-00')
1.234

Note that this field only works with Date and Timestamp input. It doesn't work with Interval type. In Spark, the number of seconds of month /day is considered as uncertain in Spark (a month contains 28~31 days, a day contains 23 ~25 hours).

Why are the changes needed?

This is useful for getting the number of seconds since 1970-01-01 00:00:00-00.
PostgreSQL also supports the same field:
https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT

The other reason is that since casting from TimestampType to Numeric Type is disallowed in ANSI mode, we need to provide a proper solution if a user has to do the casting.

Does this PR introduce any user-facing change?

Yes, a new field "EPOCH" for datetime function extract/date_part

How was this patch tested?

Unit test

[MaxGekk]

hmm, I added EPOCH in the PR #25410 but it was removed by #28284 . What is the reason to add it back again? cc @yaooqinn@cloud-fan@dongjoon-hyun

[gengliangwang]

The other reason is that since casting from TimestampType to Numeric Type is disallowed in ANSI mode, we need to provide a proper solution if a user has to do the casting.

@MaxGekk This is my major motivation. Actually, I didn't know we had it before and got removed.

[cloud-fan]

Because we thought it's not commonly used. But @gengliangwang gave a good point that we need a replacement for casting datetime to numbers.

[yaooqinn]

I checked the chat history with @cloud-fan, the reasons that we agreed for deleting EPOCH are 1) EPOCH is non-ANSI 2) EPOCH cannot express the meaning of extract comparing with other patterns. 3) supporters are few.

[cloud-fan]

ah yea, it's not really extract. Can we follow big query to add 3 functions: UNIX_SECONDS, UNIX_MILLIS, UNIX_MICROS. See https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36030/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36030/

[MaxGekk]

@gengliangwang If you would like to be compatible to PostgreSQL, you need to take the removed implementation:

/** * Returns the number of seconds with fractional part in microsecond precision * since 1970-01-01 00:00:00 local time.*/defgetEpoch(timestamp: SQLTimestamp, zoneId: ZoneId):Decimal= {
valoffset=SECONDS.toMicros(
zoneId.getRules.getOffset(microsToInstant(timestamp)).getTotalSeconds)
valsinceEpoch= timestamp + offset
Decimal(sinceEpoch, 20, 6)
}

PostgreSQL takes seconds since the local epoch 1970-01-01 00:00:00-00 but your implementation calculates seconds since 1970-01-01 00:00:00-00Z (in UTC time zone).

[gengliangwang]

+1, thanks for pointing it out 👍

[SparkQA]

Test build #131424 has finished for PR 30445 at commit 6a5f31b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

Close this since we already have #30566