[SPARK-17251][SQL] Improve OuterReference to be NamedExpression

[dongjoon-hyun]

What changes were proposed in this pull request?

Currently, OuterReference is not NamedExpression. So, it raises 'ClassCastExceptionwhen it used in projection lists of IN correlated subqueries. This PR aims to support that by makingOuterReferenceasNamedExpression` to show correct error messages.

scala> sql("CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a)")
scala> sql("CREATE TEMPORARY VIEW t2 AS SELECT * FROM VALUES 1 AS t2(b)")
scala> sql("SELECT a FROM t1 WHERE a IN (SELECT a FROM t2)").show
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.OuterReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression

How was this patch tested?

Pass the Jenkins test with new test cases.

[SparkQA]

Test build #69178 has finished for PR 16015 at commit 9d965e7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class OuterReference(e: NamedExpression)

[dongjoon-hyun]

The only one test failure is irrelevant to this PR.

[info] - set spark.sql.warehouse.dir *** FAILED *** (5 minutes, 0 seconds)
[info] Timeout of './bin/spark-submit' '--class' 'org.apache.spark.sql.hive.SetWarehouseLocationTest' '--name' 'SetSparkWarehouseLocationTest' '--master' 'local-cluster[2,1,1024]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--driver-java-options' '-Dderby.system.durability=test' 'file:/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-27a1c717-99bc-44c6-8af7-710c8440c14d/testJar-1480135147576.jar' See the log4j logs for more detail.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #69181 has started for PR 16015 at commit 9d965e7.

[dongjoon-hyun]

Unknown Jenkins failure.

Traceback (most recent call last):
File "./dev/run-tests-jenkins.py", line 232, in <module>
main()
File "./dev/run-tests-jenkins.py", line 219, in main
test_result_code, test_result_note = run_tests(tests_timeout)
File "./dev/run-tests-jenkins.py", line 140, in run_tests
test_result_note = ' * This patch **fails %s**.' % failure_note_by_errcode[test_result_code]
KeyError: -9

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #69182 has finished for PR 16015 at commit 9d965e7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class OuterReference(e: NamedExpression)

[hvanhovell]

Minor: Maybe we should update this error. Something like this: Expressions referencing the outer query are not supported outside of WHERE/HAVING clauses

[dongjoon-hyun]

Thank you for review, @hvanhovell . Sure.

[hvanhovell]

LGTM pending jenkins.

[dongjoon-hyun]

Thank you, @hvanhovell !

[SparkQA]

Test build #69192 has finished for PR 16015 at commit 007ab04.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

Merging to master/2.1/2.0. Thanks!

[dongjoon-hyun]

Thank you for merging, @hvanhovell !