[SPARK-12991] [SQL] Establish a link between SparkPlan and LogicalPlan nodes

[mbautin]

This is a prerequisite for reusing RDDs corresponding to shared query
fragments between different Spark SQL queries, which helps improve
performance significantly on many analytical workloads even without
explicitly caching any tables.

[cloud-fan]

Did you link to the corrected JIRA ticket? And what's your whole design? This is a big change and we need to discuss if it worth.

[SparkQA]

Test build #50614 has finished for PR 11036 at commit b79fd07.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mbautin]

Sorry -- linked to the wrong JIRA ticket. Should be https://issues.apache.org/jira/browse/SPARK-12991.

[mbautin]

@cloud-fan : the whole design for the feature this is needed for (query fragment RDD reuse) is at https://issues.apache.org/jira/browse/SPARK-11838, but this seems to be the only part that cannot be done without modifying the Spark SQL code, because we need to find logical plans corresponding to generated RDDs somehow.

[rxin]

This is not going to work at all with whole-stage codgen, in which we collapse all pipelinable operators into a single generated function.

[mbautin]

Even in that case, we could still obtain RDDs corresponding to SparkPlan nodes at stage boundaries, right? We would still find that useful in our query workload.

[rxin]

How would the change here help you with that?

[SparkQA]

Test build #51606 has finished for PR 11036 at commit b79fd07.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.