Skip to content

[SQL][SPARK-16888] Implements eval method for expression AssertNotNull - #14486

Closed
clockfly wants to merge 1 commit into
apache:masterfrom
clockfly:assertnotnull_eval
Closed

[SQL][SPARK-16888] Implements eval method for expression AssertNotNull#14486
clockfly wants to merge 1 commit into
apache:masterfrom
clockfly:assertnotnull_eval

Conversation

@clockfly

@clockflyclockfly commented Aug 3, 2016

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Implements eval() method for expression AssertNotNull so that we can convert local projection on LocalRelation to another LocalRelation.

Before change:

scala> import org.apache.spark.sql.catalyst.dsl.expressions._
scala> import org.apache.spark.sql.catalyst.expressions.objects.AssertNotNull
scala> import org.apache.spark.sql.Column
scala> case class A(a: Int)
scala> Seq((A(1),2)).toDS().select(new Column(AssertNotNull("_1".attr, Nil))).explain
java.lang.UnsupportedOperationException: Only code-generated evaluation is supported.
at org.apache.spark.sql.catalyst.expressions.objects.AssertNotNull.eval(objects.scala:850)
...

After the change:

scala> Seq((A(1),2)).toDS().select(new Column(AssertNotNull("_1".attr, Nil))).explain(true)
== Parsed Logical Plan ==
'Project [assertnotnull('_1) AS assertnotnull(_1)#5]
+- LocalRelation [_1#2, _2#3]
== Analyzed Logical Plan ==
assertnotnull(_1): struct<a:int>
Project [assertnotnull(_1#2) AS assertnotnull(_1)#5]
+- LocalRelation [_1#2, _2#3]
== Optimized Logical Plan ==
LocalRelation [assertnotnull(_1)#5]
== Physical Plan ==
LocalTableScan [assertnotnull(_1)#5]

How was this patch tested?

Unit test.

@SparkQA

Copy link
Copy Markdown

Test build #63189 has finished for PR 14486 at commit 595c6c8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

Copy link
Copy Markdown
Contributor

LGTM, can you create a JIRA for it?

@clockflyclockfly changed the title [SQL][MINOR] Implements eval method for expression AssertNotNull[SQL][SPARK-16888] Implements eval method for expression AssertNotNullAug 4, 2016
@cloud-fan

Copy link
Copy Markdown
Contributor

thanks, merging to master!

@asfgitasfgit closed this in 27e815cAug 4, 2016
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@clockfly@SparkQA@cloud-fan