[SPARK-22822][TEST] Basic tests for WindowFrameCoercion and DecimalPrecision by wangyum · Pull Request #20008 · apache/spark

wangyum · 2017-12-18T11:05:19Z

What changes were proposed in this pull request?

Test Coverage for WindowFrameCoercion and DecimalPrecision, this is a Sub-tasks for SPARK-22722.

How was this patch tested?

N/A

SparkQA · 2017-12-18T15:03:20Z

Test build #85058 has finished for PR 20008 at commit 05acae3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-18T17:24:26Z

+SELECT cast(1 as tinyint) + cast(1 as decimal(20, 1)) FROM t;
+SELECT cast(1 as tinyint) + cast(1 as decimal(21, 1)) FROM t;
+SELECT cast(1 as tinyint) + cast(1 as decimal(38, 1)) FROM t;
+SELECT cast(1 as tinyint) + cast(1 as decimal(39, 1)) FROM t;


Do we need all these cases? Could you simplify them?

How about only these 4 decimals: DECIMAL(3, 0), DECIMAL(5, 0), DECIMAL(10, 0) and DECIMAL(20, 0).
spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala
Lines 54 to 57 in 00d176d
* - BYTE gets turned into DECIMAL(3, 0)
* - SHORT gets turned into DECIMAL(5, 0)
* - INT gets turned into DECIMAL(10, 0)
* - LONG gets turned into DECIMAL(20, 0)

gatorsmile · 2017-12-19T08:16:26Z

functionArgumentConversion.sql has too many combinations. Let us get rid of this one.

SparkQA · 2017-12-19T17:42:00Z

Test build #85111 has finished for PR 20008 at commit 3ff5c9b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-12-20T01:40:47Z

 SELECT array(cast(1 as tinyint), cast(1 as decimal(10, 0))) FROM t;
 SELECT array(cast(1 as tinyint), cast(1 as string)) FROM t;
-SELECT array(cast(1 as tinyint), cast('1' as binary)) FROM t;
+SELECT size(array(cast(1 as tinyint), cast('1' as binary))) FROM t;


Replace array(cast(1 as tinyint), cast('1' as binary)) with size(array(cast(1 as tinyint), cast('1' as binary))) to avoid binary type with collection error.

SparkQA · 2017-12-20T04:20:08Z

Test build #85143 has finished for PR 20008 at commit 16f0a99.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-20T04:25:38Z

functionArgumentConversion.sql has too many combinations. Let us get rid of this one?

SparkQA · 2017-12-20T08:05:02Z

Test build #85156 has finished for PR 20008 at commit b4c5339.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-12-20T08:07:48Z

retest this, please.

SparkQA · 2017-12-20T10:18:49Z

Test build #85162 has finished for PR 20008 at commit b4c5339.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-12-20T12:58:28Z

 TypeCheckFailure(
 s"The data type of the $location bound '${e.dataType} does not match " +
- s"the expected data type '${frameType.inputType}'.")
+ s"the expected data type '${frameType.inputType.simpleString}'.")


Otherwise the result is:
cannot resolve 'RANGE BETWEEN CURRENT ROW AND CAST(1 AS STRING) FOLLOWING' due to data type mismatch: The data type of the upper bound 'StringType does not match the expected data type 'org.apache.spark.sql.types.TypeCollection@7ff36201'.; line 1 pos 21

SparkQA · 2017-12-20T15:00:19Z

Test build #85186 has finished for PR 20008 at commit 19bcca1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-20T15:15:30Z

Test build #85187 has finished for PR 20008 at commit 3291339.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-21T01:07:06Z

Test build #85215 has finished for PR 20008 at commit a49d1cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-12-21T01:36:53Z

Test build #85217 has finished for PR 20008 at commit ec07bc2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-12-21T04:45:46Z

retest this please

SparkQA · 2017-12-21T07:13:37Z

Test build #85233 has finished for PR 20008 at commit ec07bc2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-12-21T17:17:14Z

The test failure is not related to this PR.

LGTM

Thanks! Merged to master.

Basic tests for FunctionArgumentConversion and DecimalPrecision
05acae3

gatorsmile reviewed Dec 18, 2017
View reviewed changes

simplify
3ff5c9b

Fix binary error
16f0a99

wangyum commented Dec 20, 2017
View reviewed changes

remove functionArgumentConversion.sql and add windowFrameCoercion.sql
b4c5339

wangyum changed the title ~~[SPARK-22822][TEST] Basic tests for FunctionArgumentConversion and DecimalPrecision~~[SPARK-22822][TEST] Basic tests for WindowFrameCoercion and DecimalPrecisionDec 20, 2017

TypeCollection should use simpleString
19bcca1

wangyum commented Dec 20, 2017
View reviewed changes

Add a single quote
3291339

wangyum added 2 commits December 21, 2017 05:57

Fix test error
a49d1cc

Fix test error
ec07bc2

asfgit closed this in 4e107fdDec 21, 2017

gatorsmile mentioned this pull request Dec 24, 2017
[SPARK-22036][SQL] Decimal multiplication with high precision/scale often returns NULL #20023
Closed

	* - BYTE gets turned into DECIMAL(3, 0)
	* - SHORT gets turned into DECIMAL(5, 0)
	* - INT gets turned into DECIMAL(10, 0)
	* - LONG gets turned into DECIMAL(20, 0)

Uh oh!

Conversation

wangyum commented Dec 18, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Dec 18, 2017

Uh oh!

gatorsmileDec 18, 2017

Choose a reason for hiding this comment

Uh oh!

wangyumDec 19, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmileDec 19, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Dec 19, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Dec 19, 2017

Uh oh!

wangyumDec 20, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 20, 2017

Uh oh!

gatorsmile commented Dec 20, 2017

Uh oh!

SparkQA commented Dec 20, 2017

Uh oh!

wangyum commented Dec 20, 2017

Uh oh!

SparkQA commented Dec 20, 2017

Uh oh!

wangyumDec 20, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 20, 2017

Uh oh!

SparkQA commented Dec 20, 2017

Uh oh!

SparkQA commented Dec 21, 2017

Uh oh!

SparkQA commented Dec 21, 2017

Uh oh!

wangyum commented Dec 21, 2017

Uh oh!

SparkQA commented Dec 21, 2017

Uh oh!

gatorsmile commented Dec 21, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wangyum commented Dec 18, 2017•
edited
Loading

gatorsmile commented Dec 19, 2017•
edited
Loading

wangyumDec 20, 2017•
edited
Loading