[SPARK-20754][SQL] Support TRUNC (number) by wangyum · Pull Request #18106 · apache/spark

wangyum · 2017-05-25T11:01:50Z

What changes were proposed in this pull request?

Move TruncDate() from datetimeExpressions.scala to misc.scala, and add support TRUNC(number), it's similar to Oracle TRUNC(number):

>SELECT TRUNC(1234567891.1234567891, 4);
1234567891.1234>SELECT TRUNC(1234567891.1234567891, -4);
1234560000>SELECT TRUNC(1234567891.1234567891, 0);
1234567891>SELECT TRUNC(1234567891.1234567891);
1234567891

The MOD and POSITION function alias will be added by follow-up PR.

How was this patch tested?

unit tests

SparkQA · 2017-05-25T12:42:49Z

Test build #77361 has finished for PR 18106 at commit a5ade70.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Trunc(data: Expression, format: Expression = Literal(0))

SparkQA · 2017-05-25T16:43:07Z

Test build #77370 has finished for PR 18106 at commit c63856b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-26T02:12:42Z

Test build #77394 has finished for PR 18106 at commit e7e6e5b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-26T04:56:49Z

Test build #77399 has finished for PR 18106 at commit 7157820.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? apache#18106 Support TRUNC (number), We should also add function alias for `MOD `and `POSITION`. `POSITION(substr IN str) `is a synonym for `LOCATE(substr,str)`. same as MySQL: https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_position ## How was this patch tested? unit tests Author: Yuming Wang <wgyumg@gmail.com> Closesapache#18206 from wangyum/SPARK-20754-mod&position.

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala sql/core/src/test/resources/sql-tests/inputs/datetime.sql sql/core/src/test/resources/sql-tests/inputs/operators.sql sql/core/src/test/resources/sql-tests/results/datetime.sql.out sql/core/src/test/resources/sql-tests/results/operators.sql.out

SparkQA · 2017-06-14T17:10:32Z

Test build #78057 has finished for PR 18106 at commit 3d92a48.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Trunc(data: Expression, format: Expression)

SparkQA · 2017-06-15T04:52:33Z

Test build #78080 has started for PR 18106 at commit b391b6a.

wangyum · 2017-06-15T08:48:16Z

Jenkins, retest this please

SparkQA · 2017-06-15T11:07:07Z

Test build #78094 has finished for PR 18106 at commit b391b6a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-06-15T11:30:40Z

cc @gatorsmile

## What changes were proposed in this pull request? apache#18106 Support TRUNC (number), We should also add function alias for `MOD `and `POSITION`. `POSITION(substr IN str) `is a synonym for `LOCATE(substr,str)`. same as MySQL: https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_position ## How was this patch tested? unit tests Author: Yuming Wang <wgyumg@gmail.com> Closesapache#18206 from wangyum/SPARK-20754-mod&position.

SparkQA · 2017-06-20T06:27:38Z

Test build #78273 has started for PR 18106 at commit 5456e61.

wangyum · 2017-06-20T21:59:29Z

Retest this please.

SparkQA · 2017-06-20T22:02:38Z

Test build #78325 has started for PR 18106 at commit 5456e61.

shaneknapp · 2017-06-21T00:38:20Z

test this please

SparkQA · 2017-06-21T00:42:39Z

Test build #78344 has started for PR 18106 at commit 5456e61.

shaneknapp · 2017-06-21T04:21:22Z

test this please

viirya · 2017-06-21T05:15:42Z

+
+ override def inputTypes: Seq[AbstractDataType] =
+ Seq(TypeCollection(DateType, DoubleType, DecimalType),
+ TypeCollection(StringType, IntegerType))


I think this might lead to wrong input types combinations such as (DoubleType, StringType) and (DateType, IntegerType)?

If we are going to have only trunc for truncating number and datetime. We should prevent wrong input types.

viirya · 2017-06-21T05:18:43Z

Is there duplicated codes between trunc(number) and trunc(date)? If no, seems to me we don't necessarily let one expression to have two different features.

viirya · 2017-06-21T05:27:12Z

Although then we can't use just one trunc function, it seems ok for me because not all databases use trunc to truncate both number and datetime.

SparkQA · 2017-06-27T12:47:01Z

Test build #78693 has finished for PR 18106 at commit f8b1f44.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2017-06-27T16:36:55Z


 @since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):


@ueshin @holdenk re: changing param name in python.

I believe this definitely breaks backward compatibility for keyword-argument usage in Python.

felixcheung · 2017-06-27T16:37:20Z

I'll add @gatorsmile since this is SQL.

wangyum · 2017-06-28T09:44:20Z

Jenkins, retest this please

SparkQA · 2017-06-28T11:23:31Z

Test build #78786 has finished for PR 18106 at commit f8b1f44.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-07-31T03:42:08Z

retest this please

SparkQA · 2017-07-31T05:49:14Z

Test build #80066 has finished for PR 18106 at commit f8b1f44.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-08-01T03:32:04Z

ping @wangyum

wangyum · 2017-08-01T10:28:04Z

I'll fix it

SparkQA · 2017-08-02T07:04:51Z

Test build #80150 has finished for PR 18106 at commit 3d40c36.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-08-02T07:07:31Z

Jenkins, retest this please

HyukjinKwon · 2017-08-02T07:16:50Z


 @since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):


@wangyum, would you mind revert this renaming? This breaks the compatibility if user script calls this by
trunc(..., format= ...) trunc(date=..., format= ...)

We can work around this with kwargs if it's important to change the name.

Yes but it brings complexity for both args and kwargs e.g., when both set, method signature in doc and etc. I wonder if it is that important.

SparkQA · 2017-08-02T09:49:48Z

Test build #80152 has finished for PR 18106 at commit 3d40c36.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-02T15:20:38Z

Test build #80159 has finished for PR 18106 at commit 931f07d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-10-27T23:47:48Z

This is close to merge. Could you resolve the conflicts? Then, I will review it. Thanks!

felixcheung · 2017-10-28T04:47:42Z

R has trunc in master/2.3 as well

# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala # sql/core/src/test/resources/sql-tests/inputs/operators.sql # sql/core/src/test/resources/sql-tests/results/operators.sql.out

-trunc

# Conflicts: # sql/core/src/test/resources/sql-tests/inputs/datetime.sql # sql/core/src/test/resources/sql-tests/results/datetime.sql.out

SparkQA · 2017-10-28T09:27:55Z

Test build #83155 has finished for PR 18106 at commit b59a2df.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2017-10-28T10:19:07Z

Test build #83158 has finished for PR 18106 at commit 679ff98.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ClearCacheCommand() extends RunnableCommand

dongjoon-hyun · 2018-09-13T17:18:53Z

@wangyum .
https://issues.apache.org/jira/browse/SPARK-20754 is resolved by you. Could you close this PR?

wangyum · 2018-09-14T03:08:17Z

@dongjoon-hyun Actually TRUNC (number) not resolved. I will fix it soon.
https://issues.apache.org/jira/browse/SPARK-23906

dongjoon-hyun · 2018-09-14T03:12:38Z

+100, @wangyum . Thanks. :)

Support TRUNC (number)
a5ade70

Fix failed tests
c63856b

wangyum added 2 commits May 26, 2017 08:09

Add test for python api
224c867

Update comment.
e7e6e5b

Fix PySpark unit test issue
7157820

wangyum mentioned this pull request Jun 5, 2017
[SPARK-20754][SQL][FOLLOWUP] Add Function Alias For MOD/POSITION. #18206
Closed

wangyum added 2 commits June 14, 2017 22:54

Refactor code.
3d92a48

Add comment.
b391b6a

Merge branch 'master' intoSPARK-20754-trunc
5456e61

viirya reviewed Jun 21, 2017
View reviewed changes

Support timestamp and string type.
f8b1f44

felixcheung reviewed Jun 27, 2017
View reviewed changes

codegen support String and Timestamp type.
3d40c36

HyukjinKwon reviewed Aug 2, 2017
View reviewed changes

Revert trunc(date, format).
931f07d

wangyum added 3 commits October 28, 2017 14:39

Merge remote-tracking branch 'origin/SPARK-20754-trunc' intoSPARK-20754
b59a2df
-trunc

Merge remote-tracking branch 'upstream/master' intoSPARK-20754-trunc
679ff98
# Conflicts: # sql/core/src/test/resources/sql-tests/inputs/datetime.sql # sql/core/src/test/resources/sql-tests/results/datetime.sql.out

wangyum closed this Sep 14, 2018

maropu mentioned this pull request Sep 18, 2018
[SPARK-23906][SQL] Add built-in UDF TRUNCATE(number) #22419
Closed

wangyum deleted the SPARK-20754-trunc branch October 8, 2019 04:25

Uh oh!

Conversation

wangyum commented May 25, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 25, 2017

Uh oh!

SparkQA commented May 25, 2017

Uh oh!

SparkQA commented May 26, 2017

Uh oh!

SparkQA commented May 26, 2017

Uh oh!

SparkQA commented Jun 14, 2017

Uh oh!

SparkQA commented Jun 15, 2017

Uh oh!

wangyum commented Jun 15, 2017

Uh oh!

SparkQA commented Jun 15, 2017

Uh oh!

wangyum commented Jun 15, 2017

Uh oh!

SparkQA commented Jun 20, 2017

Uh oh!

wangyum commented Jun 20, 2017

Uh oh!

SparkQA commented Jun 20, 2017

Uh oh!

shaneknapp commented Jun 21, 2017

Uh oh!

SparkQA commented Jun 21, 2017

Uh oh!

shaneknapp commented Jun 21, 2017

Uh oh!

viiryaJun 21, 2017

Choose a reason for hiding this comment

Uh oh!

viiryaJun 21, 2017

Choose a reason for hiding this comment

Uh oh!

viirya commented Jun 21, 2017

Uh oh!

viirya commented Jun 21, 2017

Uh oh!

SparkQA commented Jun 27, 2017

Uh oh!

felixcheungJun 27, 2017

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 27, 2017

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Jun 27, 2017

Uh oh!

wangyum commented Jun 28, 2017

Uh oh!

SparkQA commented Jun 28, 2017

Uh oh!

gatorsmile commented Jul 31, 2017

Uh oh!

SparkQA commented Jul 31, 2017

Uh oh!

gatorsmile commented Aug 1, 2017

Uh oh!

wangyum commented Aug 1, 2017

Uh oh!

SparkQA commented Aug 2, 2017

Uh oh!

wangyum commented Aug 2, 2017

Uh oh!

HyukjinKwonAug 2, 2017

Choose a reason for hiding this comment

Uh oh!

holdenkAug 2, 2017

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonAug 2, 2017

wangyum commented May 25, 2017•
edited
Loading