[SPARK-8371][SQL] improve unit test for MaxOf and MinOf and fix bugs by cloud-fan · Pull Request #6825 · apache/spark

cloud-fan · 2015-06-15T08:16:31Z

a follow up of #6813

cloud-fan · 2015-06-15T08:24:08Z

This test will fail at checkEvaluationWithGeneratedProjection for hash code not equal. I'm not sure why we want to compare the hash code, and I looked into the hashCode implementation of GenericRow and Row, they are different. I'm trying to figure it out.

cloud-fan · 2015-06-15T08:24:27Z

cc @rxin @davies

cloud-fan · 2015-06-15T08:26:05Z

The origin code is not right as super here refers to BinaryArithmetic, not Expression that we wanted.

yes,i think you are right. because their symbol is empty.

SparkQA · 2015-06-15T09:25:59Z

Test build #34923 has finished for PR 6825 at commit cd51a7d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-15T10:50:55Z

Test build #34933 has finished for PR 6825 at commit 03021bc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

lianhuiwang · 2015-06-15T11:24:33Z

i think at first it needs to check whether eval1 or eval2 isNull. and then calculate primitive.

the code result is inserted after null check.

cloud-fan · 2015-06-15T14:04:28Z

retest it please.

davies · 2015-06-16T02:53:23Z

@cloud-fan There are some changes related to Max/Min in #6726 , is it useful for you? It will be good if you could merge them here.

SparkQA · 2015-06-16T16:20:38Z

Test build #34987 has finished for PR 6825 at commit f153380.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class IsNull(child: Expression) extends UnaryExpression with Predicate
- case class IsNotNull(child: Expression) extends UnaryExpression with Predicate

davies · 2015-06-17T03:49:31Z

All expressions only work with InternalRow, which only contain internal types (int, long, UTF8String), no Date or Timestamp. so we don't need to test Date/Timestamp, or use DateUtils.fromDate() to do the conversion before calling expression on them.

cloud-fan · 2015-06-17T15:02:16Z

This test will fail at checkEvaluationWithGeneratedProjection for hash code not equal. I'm not sure why we want to compare the hash code, and I looked into the hashCode implementation of GenericRow and Row, they are different. Does anyone know why we want to compare the hash code there?

SparkQA · 2015-06-17T15:14:10Z

Test build #35044 has finished for PR 6825 at commit 97066a6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2015-06-17T15:48:02Z

@davies , we need the conversion here, as we use String instead of UTF8String at many tests...

Sound reasonable, it's anonying to have so many UTF8String.fromString in test cases.

SparkQA · 2015-06-17T17:40:24Z

Test build #35045 has finished for PR 6825 at commit 8c3fbde.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-06-23T19:07:35Z

@cloud-fan Now #6876 is merged, could you update this PR?

SparkQA · 2015-06-24T06:06:14Z

Test build #35636 has finished for PR 6825 at commit 43170cc.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-06-24T06:10:19Z

Good catch!

davies · 2015-06-24T06:11:22Z

LGTM, the failed PySpark test is not related, merging this into master, thanks!

SparkQA · 2015-06-24T08:02:16Z

Test build #955 has finished for PR 6825 at commit 43170cc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan reviewed Jun 15, 2015
View reviewed changes

lianhuiwang reviewed Jun 15, 2015
View reviewed changes

cloud-fan force-pushed the cg branch from 03021bc to f153380Compare June 16, 2015 14:36

davies reviewed Jun 17, 2015
View reviewed changes

cloud-fan force-pushed the cg branch from f153380 to 97066a6Compare June 17, 2015 14:57

cloud-fan reviewed Jun 17, 2015
View reviewed changes

cloud-fan mentioned this pull request Jun 18, 2015
[WIP] remove expensive api from InternalRow #6869
Closed

fix bugs in code gen
43170cc

cloud-fan force-pushed the cg branch from 3c9aa51 to 43170ccCompare June 24, 2015 04:43

davies reviewed Jun 24, 2015
View reviewed changes

Comment thread...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
daviesJun 24, 2015
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch!

asfgit closed this in 09fcf96Jun 24, 2015

cloud-fan deleted the cg branch June 24, 2015 06:32

Uh oh!

Conversation

cloud-fan commented Jun 15, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jun 15, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 15, 2015

Uh oh!

SparkQA commented Jun 15, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jun 15, 2015

Uh oh!

davies commented Jun 16, 2015

Uh oh!

SparkQA commented Jun 16, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 17, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 17, 2015

Uh oh!

davies commented Jun 23, 2015

Uh oh!

SparkQA commented Jun 24, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davies commented Jun 24, 2015

Uh oh!

SparkQA commented Jun 24, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants