[SPARK-12957][SQL] Initial support for constraint propagation in SparkSQL

[sameeragarwal]

Based on the semantics of a query, we can derive a number of data constraints on output of each (logical or physical) operator. For instance, if a filter defines ‘a > 10, we know that the output data of this filter satisfies 2 constraints:

1. ‘a > 10
2. isNotNull(‘a)

This PR proposes a possible way of keeping track of these constraints and propagating them in the logical plan, which can then help us build more advanced optimizations (such as pruning redundant filters, optimizing joins, among others). We define constraints as a set of (implicitly conjunctive) expressions. For e.g., if a filter operator has constraints = Set(‘a > 10, ‘b < 100), it’s implied that the outputs satisfy both individual constraints (i.e., ‘a > 10 AND ‘b < 100).

Design Document: https://docs.google.com/a/databricks.com/document/d/1WQRgDurUBV9Y6CWOBS75PQIqJwT-6WftVa18xzm7nCo/edit?usp=sharing

[SparkQA]

Test build #49770 has finished for PR 10844 at commit f7251dd.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49775 has finished for PR 10844 at commit 04ff99a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

Could you add the id of the JIRA ticket to the titlle?

Could you also add a description, explaining why we want this? Seems cool though!

[sameeragarwal]

@hvanhovell added, thanks!

[gatorsmile]

@sameeragarwal I just saw it. Thank you! I will do the code changes after this is merged.

[SparkQA]

Test build #50083 has finished for PR 10844 at commit 7fb2f9c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50161 has finished for PR 10844 at commit f15ef96.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

protected?

Also I'm not sure I get the scala doc. Maybe getReleventContraints is a better name? It is taking the constraints and removing those that don't apply anymore because we removed columns right?

[SparkQA]

Test build #50371 has finished for PR 10844 at commit 53be837.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sameeragarwal]

Thanks @marmbrus, all comments addressed!

[SparkQA]

Test build #50404 has finished for PR 10844 at commit 8c6bb70.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sameeragarwal]

test this please

[marmbrus]

I would consider making this getRelevantConstraints(validConstraints) so that each implementor of validContraints does't have to remember to do the filter / canonicalization. They can just focus on augmenting or passing through constraints from children based on the operators semantics.

[sameeragarwal]

Great idea, added.

[SparkQA]

Test build #50417 has finished for PR 10844 at commit 8c6bb70.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sameeragarwal]

Thanks @marmbrus, all comments addressed!

[SparkQA]

Test build #50429 has finished for PR 10844 at commit 302444f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

I think this was a merging mistake as its duplicated with the method below.

[sameeragarwal]

oops, fixed!

[sameeragarwal]

@marmbrus comments addressed!

[sameeragarwal]

test this please

[SparkQA]

Test build #50518 has finished for PR 10844 at commit b52742a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Looks like we do not need PredicateHelper at here? Maybe it is better to just with PredicateHelper for Filter and Join?

[sameeragarwal]

Sounds good, added this only for filter and join that needed splitConjunctivePredicates

[sameeragarwal]

comments addressed!

[SparkQA]

Test build #50603 has finished for PR 10844 at commit 2bd2735.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class QueryPlan[PlanType <: TreeNode[PlanType]] extends TreeNode[PlanType]
  • abstract class UnaryNode extends LogicalPlan
  • case class Filter(condition: Expression, child: LogicalPlan)

[marmbrus]

Thanks! Merging to master.