[SPARK-7142][SQL]: Minor enhancement to BooleanSimplification Optimizer rule

[saucam]

Use these in the optimizer as well:

 A and (not(A) or B) => A and B
not(A and B) => not(A) or not(B)
not(A or B) => not(A) and not(B)

[SparkQA]

Test build #30953 has finished for PR 5700 at commit 9e1f8dd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class IDF extends Estimator[IDFModel] with IDFBase
• This patch adds the following new dependencies:
  • tachyon-0.6.4.jar
  • tachyon-client-0.6.4.jar
• This patch removes the following dependencies:
  • tachyon-0.5.0.jar
  • tachyon-client-0.5.0.jar

[cloud-fan]

How these 2 rules optimize execution?

[saucam]

So for example the filter is not(Or(left, r)) , where r might be some filter on a partitioned column like part>=12 , in the present case this filter cannot be pushed down, since while evaluating we will encounter reference of partitioned column, whereas if this rule is applied we get And(not(l), part<12) and then not(l) might be pushed down since now splitting into conjunctive predicates is possible.

[cloud-fan]

case Not(Or(l, r))? Seems you miss the Not...

[saucam]

This is inside a case match :

 case not @ Not(exp) => exp match {
....
....
case Or(l, r) => And(Not(l), Not(r))
}

[cloud-fan]

Changing Not(And(a, b)) to And(Not(a), Not(b)) may not always be an optimization. It do helps to push down filters for datasource, but I think #8200 is more reasonable since it only do this optimization in datasource part.

cc @marmbrus

[saucam]

@cloud-fan could you please explain a bit more when and how converting to "And" may not be an optimization ? I was wondering would it actually result in any kind of performance hit ? Also could you tell how #8200 is more reasonable ?

[cloud-fan]

At first I thought this optimization should be only used in datasource as it may cause performance regression for normal cases(see these comments). However after an offline discussion with @marmbrus , this optimization is pretty standard and generally good, so let's keep it in Optimizer :)

[marmbrus]

Thanks for working on this! Can you please add test cases for these optimizations?

[saucam]

added test cases

[SparkQA]

Test build #42249 has finished for PR 5700 at commit de2dbdc.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42243 has finished for PR 5700 at commit 7ada093.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42257 has finished for PR 5700 at commit 2dbfba6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Thanks, merging to master.

[saucam]

thanks @marmbrus :)

[cloud-fan]

Do we need fastEquals here? Not(l) and l1 will never be a same reference and we always fallback to normal equality check. I think just here == here is more reasonable.

[marmbrus]

+1