[SPARK-12594] [SQL] Outer Join Elimination by Filter Conditions

[gatorsmile]

Conversion of outer joins, if the predicates in filter conditions can restrict the result sets so that all null-supplying rows are eliminated.

• full outer -> inner if both sides have such predicates
• left outer -> inner if the right side has such predicates
• right outer -> inner if the left side has such predicates
• full outer -> left outer if only the left side has such predicates
• full outer -> right outer if only the right side has such predicates

If applicable, this can greatly improve the performance, since outer join is much slower than inner join, full outer join is much slower than left/right outer join.

The original PR is #10542

[SparkQA]

Test build #48633 has finished for PR 10567 at commit d516ed4.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48634 has finished for PR 10567 at commit 3c9e965.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Can you document the arguments here?

@sameeragarwal it would be great if we could replace this function with your more general null propagation information

/cc @yhuai

[sameeragarwal]

@gatorsmile now that we propagate IsNotNull constraints in the logical plan, you should be able to eliminate outer joins by simply looking into the constraints of the parent filter operator. I believe something along the lines this should work:

objectOuterJoinEliminationextendsRule[LogicalPlan] withPredicateHelper {
privatedefbuildNewJoin(filter: Filter, join: Join):Join= {
valleftHasNonNullPredicate= filter.constraints.filter(_.isInstanceOf[IsNotNull])
.exists(expr => join.left.outputSet.intersect(expr.references).nonEmpty)
valrightHasNonNullPredicate= filter.constraints.filter(_.isInstanceOf[IsNotNull])
.exists(expr => join.right.outputSet.intersect(expr.references).nonEmpty)
join.joinType match {
caseRightOuterif leftHasNonNullPredicate =>Join(join.left, join.right, Inner, join.condition)
caseLeftOuterif rightHasNonNullPredicate =>Join(join.left, join.right, Inner, join.condition)
caseFullOuterif leftHasNonNullPredicate && rightHasNonNullPredicate =>Join(join.left, join.right, Inner, join.condition)
caseFullOuterif leftHasNonNullPredicate =>Join(join.left, join.right, LeftOuter, join.condition)
caseFullOuterif rightHasNonNullPredicate =>Join(join.left, join.right, RightOuter, join.condition)
case _ =>
join
}
}
defapply(plan: LogicalPlan):LogicalPlan= plan transform {
case f @Filter(condition, j@Join(_, _, RightOuter|LeftOuter|FullOuter, _)) =>Filter(condition, buildNewJoin(f, j))
}
}

[gatorsmile]

Thank you very much! Will do the changes.

[SparkQA]

Test build #51088 has finished for PR 10567 at commit 560f2d0.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51098 has started for PR 10567 at commit e7fa63f.

[shaneknapp]

jenkins, test this please

[SparkQA]

Test build #51115 has finished for PR 10567 at commit e7fa63f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

I think we can support all the expressions that will return null or false, if the inputs are null.

[gatorsmile]

With your changes, we can support all the expressions whose attributes are all from left or right, no matter how complicated they are!!!

[gatorsmile]

With @sameeragarwal changes, we can support the expressions containing both left and right attributes but the types are limited to EqualTo, GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual, and EqualNullSafe.

We need both!

[SparkQA]

Test build #51499 has finished for PR 10567 at commit 11b3214.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51520 has finished for PR 10567 at commit 6977fdf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

return the original f if it's not changed

[gatorsmile]

Thanks! will do.

[SparkQA]

Test build #51570 has finished for PR 10567 at commit cc0262c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

It's better to return JoinType from buildNewJoin

[gatorsmile]

Sure, will do.

[davies]

LGTM, pending tests.

[SparkQA]

Test build #51587 has finished for PR 10567 at commit 82357e0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

Merging this into master, thanks!