Skip to content

[SPARK-28481][SQL] More expressions should extend NullIntolerant - #28626

Closed
wangyum wants to merge 8 commits into
apache:masterfrom
wangyum:SPARK-28481
Closed

[SPARK-28481][SQL] More expressions should extend NullIntolerant#28626
wangyum wants to merge 8 commits into
apache:masterfrom
wangyum:SPARK-28481

Conversation

@wangyum

@wangyumwangyum commented May 24, 2020•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

  1. Make more expressions extend NullIntolerant.
  2. Add a checker(in ExpressionInfoSuite) to identify whether the expression is NullIntolerant.

Why are the changes needed?

Avoid skew join if the join column has many null values and can improve query performance. For examples:

CREATETABLEt1(c1 string, c2 string) USING parquet;
CREATETABLEt2(c1 string, c2 string) USING parquet;
EXPLAIN SELECT t1.*FROM t1 JOIN t2 ONupper(t1.c1) =upper(t2.c1);

Before and after this PR:

== Physical Plan ==*(2) Project [c1#5, c2#6]+-*(2) BroadcastHashJoin [upper(c1#5)], [upper(c1#7)], Inner, BuildLeft
:- BroadcastExchange HashedRelationBroadcastMode(List(upper(input[0, string, true]))), [id=#41]
: +-*(1) ColumnarToRow
: +- FileScan parquet default.t1[c1#5,c2#6]+-*(2) ColumnarToRow
+- FileScan parquet default.t2[c1#7] == Physical Plan ==*(2) Project [c1#5, c2#6]+-*(2) BroadcastHashJoin [upper(c1#5)], [upper(c1#7)], Inner, BuildRight
:-*(2) Project [c1#5, c2#6]
: +-*(2) Filter isnotnull(c1#5)
: +-*(2) ColumnarToRow
: +- FileScan parquet default.t1[c1#5,c2#6]+- BroadcastExchange HashedRelationBroadcastMode(List(upper(input[0, string, true]))), [id=#59]+-*(1) Project [c1#7]+-*(1) Filter isnotnull(c1#7)+-*(1) ColumnarToRow
+- FileScan parquet default.t2[c1#7]

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

@SparkQA

SparkQA commented May 24, 2020

Copy link
Copy Markdown

Test build #123041 has finished for PR 28626 at commit c3625ca.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum

wangyum commented May 24, 2020

Copy link
Copy Markdown
MemberAuthor

retest this please.

maropu
maropu reviewed May 24, 2020
View reviewed changes

@maropumaropu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @JoshRosen

Comment thread...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullIntolerantCheckerSuite.scala Outdated
Comment thread...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullIntolerantCheckerSuite.scala Outdated
Comment thread...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullIntolerantCheckerSuite.scala Outdated
Comment thread...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/NullIntolerantCheckerSuite.scala Outdated
@SparkQA

SparkQA commented May 24, 2020

Copy link
Copy Markdown

Test build #123053 has finished for PR 28626 at commit c3625ca.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented May 24, 2020

Copy link
Copy Markdown

Test build #123059 has finished for PR 28626 at commit 7fe3490.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

maropu
maropu reviewed May 25, 2020
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
@maropu

maropu commented May 25, 2020

Copy link
Copy Markdown
Member

also cc: @viirya

maropu
maropu reviewed May 25, 2020
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
maropu
maropu reviewed May 25, 2020
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala Outdated
wangyum added 2 commits May 26, 2020 11:42
@wangyum
Merge remote-tracking branch 'upstream/master' intoSPARK-28481
2c943cc
# Conflicts:
#	sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
@wangyum
fix
b127c41
@SparkQA

SparkQA commented May 26, 2020

Copy link
Copy Markdown

Test build #123105 has finished for PR 28626 at commit b127c41.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya

viirya commented May 26, 2020

Copy link
Copy Markdown
Member

retest this please

@SparkQA

SparkQA commented May 26, 2020

Copy link
Copy Markdown

Test build #123113 has finished for PR 28626 at commit b127c41.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented May 26, 2020

Copy link
Copy Markdown

Test build #123124 has finished for PR 28626 at commit 82e97e3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum

wangyum commented May 28, 2020

Copy link
Copy Markdown
MemberAuthor

@gatorsmile@cloud-fan

cloud-fan
cloud-fan reviewed May 28, 2020
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala

final override def eval(input: InternalRow): Any = {
val input2 = right.eval(input)
if (input2 == null || input2 == 0) {

@cloud-fancloud-fanMay 28, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I see the difference now. Previously we can skip evaluating input1 if input2 is 0. Can we change it back and add comment to explain it? sorry for the back and forth!

maropu
maropu approved these changes May 28, 2020
View reviewed changes

@maropumaropu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay. Thanks for the updates.

@SparkQA

SparkQA commented May 28, 2020

Copy link
Copy Markdown

Test build #123233 has finished for PR 28626 at commit 1bdff95.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented May 28, 2020

Copy link
Copy Markdown

Test build #123237 has finished for PR 28626 at commit 265abd3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented May 29, 2020

Copy link
Copy Markdown
Contributor

thanks, merging to master!

@wangyum
wangyum deleted the SPARK-28481 branch May 29, 2020 07:39
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fancloud-fancloud-fan left review comments
@maropumaropumaropu approved these changes

Assignees

No one assigned

Labels

BUILDINFRASQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wangyum@SparkQA@maropu@viirya@cloud-fan