Skip to content

[SPARK-23914][SQL][follow-up] refactor ArrayUnion - #21937

Closed
kiszk wants to merge 4 commits into
apache:masterfrom
kiszk:SPARK-23914-follow
Closed

[SPARK-23914][SQL][follow-up] refactor ArrayUnion#21937
kiszk wants to merge 4 commits into
apache:masterfrom
kiszk:SPARK-23914-follow

Conversation

@kiszk

@kiszkkiszk commented Jul 31, 2018

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR refactors ArrayUnion based on this suggestion.

  1. Generate optimized code for all of the primitive types except boolean
  2. Generate code using ArrayBuilder or ArrayBuffer
  3. Leave only a generic path in the interpreted path

How was this patch tested?

Existing tests

@holdensmagicalunicorn

holdensmagicalunicorn commented Jul 31, 2018

Copy link
Copy Markdown

@kiszk, thanks! I am a bot who has found some folks who might be able to help with the review:@ueshin, @rxin and @colorant

@SparkQA

SparkQA commented Jul 31, 2018

Copy link
Copy Markdown

Test build #93840 has finished for PR 21937 at commit 738a4a0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class OpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](
  • sealed class Hasher[@specialized(Long, Int, Double, Float) T] extends Serializable
  • class DoubleHasher extends Hasher[Double]
  • class FloatHasher extends Hasher[Float]
  • case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike

@kiszk
kiszkforce-pushed the SPARK-23914-follow branch from 738a4a0 to f48dde0CompareAugust 4, 2018 07:28
@SparkQA

SparkQA commented Aug 4, 2018

Copy link
Copy Markdown

Test build #94208 has finished for PR 21937 at commit f48dde0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

ueshin
ueshin reviewed Aug 6, 2018
View reviewed changes
Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Outdated

@ueshinueshinAug 6, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: new $arrayBuilderClass() should work?

Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Outdated

@ueshinueshinAug 6, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: classOf[mutable.ArrayBuilder[_]].getName?

@ueshin

ueshin commented Aug 6, 2018

Copy link
Copy Markdown
Member

I'll revisit after the conflict is fixed.

@kiszk

kiszk commented Aug 6, 2018

Copy link
Copy Markdown
MemberAuthor

I see. Now, I am rebasing and resolving conflicts.

@kiszk
kiszkforce-pushed the SPARK-23914-follow branch from f48dde0 to 93ec9ecCompareAugust 6, 2018 15:35
@kiszkkiszk changed the title [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion[SPARK-23914][SQL][follow-up] refactor ArrayUnionAug 6, 2018
ueshin
ueshin reviewed Aug 6, 2018
View reviewed changes
Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$hsTypeName()"
val hashSet = ctx.freshName("hashSet")
val arrayBuilder = classOf[mutable.ArrayBuilder[_]].getName
val arrayBuilderClass = s"$arrayBuilder$$of$ptName"

@ueshinueshinAug 6, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not needed any more?

@kiszkkiszkAug 7, 2018

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we still need this to create an intermediate result array. The array is allocated at L3907 as other functions do.

@ueshinueshinAug 7, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry, I misread.

@kiszkkiszk mentioned this pull request Aug 6, 2018
Closed
@SparkQA

SparkQA commented Aug 6, 2018

Copy link
Copy Markdown

Test build #94290 has finished for PR 21937 at commit 93ec9ec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Aug 6, 2018

Copy link
Copy Markdown

Test build #94304 has finished for PR 21937 at commit 26c1f8a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ueshin

ueshin commented Aug 7, 2018

Copy link
Copy Markdown
Member

LGTM.

@ueshin

ueshin commented Aug 7, 2018

Copy link
Copy Markdown
Member

Thanks! merging to master.

@asfgitasfgit closed this in 4446a0bAug 7, 2018
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ueshinueshinueshin left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kiszk@holdensmagicalunicorn@SparkQA@ueshin