[SPARK-13495][SQL] Add Null Filters in the query plan for Filters/Joins based on their data constraints

[sameeragarwal]

What changes were proposed in this pull request?

This PR adds an optimizer rule to eliminate reading (unnecessary) NULL values if they are not required for correctness by inserting isNotNull filters is the query plan. These filters are currently inserted beneath existing Filter and Join operators and are inferred based on their data constraints.

Note: While this optimization is applicable to all types of join, it primarily benefits Inner and LeftSemi joins.

How was this patch tested?

1. Added a new NullFilteringSuite that tests for IsNotNull filters in the query plan for joins and filters. Also, tests interaction with the CombineFilters optimizer rules.
2. Test generated ExpressionTrees via OrcFilterSuite
3. Test filter source pushdown logic via SimpleTextHadoopFsRelationSuite

cc @yhuai@nongli

[SparkQA]

Test build #51980 has finished for PR 11372 at commit 06d74da.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51994 has finished for PR 11372 at commit 2345075.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

I am wondering if the optimizer is the right place for this rule. My main concern is that if we can preserve this ordering through the rest of query compilation. Will it be better to do it inside the physical Filter operator (just before we start to generate the code)?

[sameeragarwal]

yes, that sounds like a good idea! Could there be any other downside of not doing it in the optimizer? /cc @nongli

[SparkQA]

Test build #52338 has finished for PR 11372 at commit 28050b3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class PlanTest extends SparkFunSuite with PredicateHelper

[SparkQA]

Test build #52383 has finished for PR 11372 at commit 2a469e8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52406 has finished for PR 11372 at commit 80dab7e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52416 has finished for PR 11372 at commit 013f97a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sameeragarwal]

test this please

[SparkQA]

Test build #52431 has finished for PR 11372 at commit 013f97a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[nongli]

"in the query plan"

[sameeragarwal]

Thanks @nongli, all comments addressed.

[SparkQA]

Test build #52494 has finished for PR 11372 at commit 31b1700.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[nongli]

LGTM