[SPARK-16995][SQL] TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr

[viirya]

What changes were proposed in this pull request?

A TreeNodeException is thrown when executing the following minimal example in Spark 2.0.

import spark.implicits._
case class test (x: Int, q: Int)
val d = Seq(1).toDF("x")
d.withColumn("q", lit(0)).as[test].groupByKey(_.x).flatMapGroups{case (x, iter) => List[Int]()}.show
d.withColumn("q", expr("0")).as[test].groupByKey(_.x).flatMapGroups{case (x, iter) => List[Int]()}.show

The problem is at FoldablePropagation. The rule will do transformExpressions on LogicalPlan. The query above contains a MapGroups which has a parameter dataAttributes:Seq[Attribute]. One attributes in dataAttributes will be transformed to an Alias(literal(0), _) in FoldablePropagation. Alias is not an Attribute and causes the error.

We can't easily detect such type inconsistency during transforming expressions. A direct approach to this problem is to skip doing FoldablePropagation on object operators as they should not contain such expressions.

How was this patch tested?

Jenkins tests.

[SparkQA]

Test build #63786 has finished for PR 14648 at commit 2e0d7d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

cc @cloud-fan

[cloud-fan]

we should follow other cases, to set stop to true

[viirya]

Ok. I will update once I return back to laptop.

[SparkQA]

Test build #63838 has finished for PR 14648 at commit 4094b43.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #63845 has finished for PR 14648 at commit 4094b43.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

After think about it more, I think the foldable propagation is still useful for object operators like DeserializeToObject, how about we specify the problematic cases here explicitly? i.e. MapGroups, FlatMapGroupsInR, CoGroup

[viirya]

You mean deserializer in DeserializeToObject can still contain foldable expressions?

[cloud-fan]

It can, if the referenced column is a literal.

[SparkQA]

Test build #63892 has finished for PR 14648 at commit 0008c3e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

why not use checkDataset?

Test build #63962 has finished for PR 14648 at commit b688abf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

thanks, merging to master and 2.0!

Thanks for review.