[SPARK-16994][SQL] PushDownPredicate should not ignore limits.

[dongjoon-hyun]

What changes were proposed in this pull request?

Currently, filter and limit are illegally permuted because PushDownPredicate optimizer pushes filters through limit. This PR fixes that.

Reported Error Scenario

scala> spark.createDataset(1 to 100).limit(10).filter($"value"%10===0).explain
==PhysicalPlan==CollectLimit10+-*Filter ((value#875%10) =0)
+-LocalTableScan [value#875]
scala> spark.createDataset(1 to 100).limit(10).filter($"value"%10===0).collect
res23:Array[Int] =Array(10, 20, 30, 40, 50, 60, 70, 80, 90, 100)

After

scala> spark.createDataset(1 to 100).limit(10).filter($"value"%10===0).explain
==PhysicalPlan==*Filter ((value#1%10) =0)
+-*GlobalLimit10+-ExchangeSinglePartition+-*LocalLimit10+-LocalTableScan [value#1]
scala> spark.createDataset(1 to 100).limit(10).filter($"value"%10===0).collect
res1:Array[Int] =Array(10)

How was this patch tested?

Pass the Jenkins with a new test case.

[gatorsmile]

Writing an end-to-end test case is not very helpful here.

You can write a new test case in the related optimizer rule.

[dongjoon-hyun]

Thank you, @gatorsmile .

[SparkQA]

Test build #63548 has finished for PR 14583 at commit aa84c8e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63581 has finished for PR 14583 at commit d23d348.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please

[SparkQA]

Test build #63589 has finished for PR 14583 at commit d23d348.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63732 has finished for PR 14583 at commit 91fb344.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @rxin .
Could you review this PR about PushDownPredicate?

[SparkQA]

Test build #63854 has finished for PR 14583 at commit c09fdc6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63986 has finished for PR 14583 at commit eea0dfe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

I'm fixing this differently here: #14713

[dongjoon-hyun]

Okay. Thanks, @rxin.