[SPARK-16994][SQL] PushDownPredicate should not ignore limits.

[dongjoon-hyun]

What changes were proposed in this pull request?

Currently, filter and limit are illegally permuted because PushDownPredicate optimizer pushes filters through limit. This PR fixes that.

Reported Error Scenario

scala> spark.createDataset(1 to 100).limit(10).filter($"value" % 10 === 0).explain
== Physical Plan ==
CollectLimit 10
+- *Filter ((value#875 % 10) = 0)
   +- LocalTableScan [value#875]

scala> spark.createDataset(1 to 100).limit(10).filter($"value" % 10 === 0).collect
res23: Array[Int] = Array(10, 20, 30, 40, 50, 60, 70, 80, 90, 100)

After

scala> spark.createDataset(1 to 100).limit(10).filter($"value" % 10 === 0).explain
== Physical Plan ==
*Filter ((value#1 % 10) = 0)
+- *GlobalLimit 10
   +- Exchange SinglePartition
      +- *LocalLimit 10
         +- LocalTableScan [value#1]

scala> spark.createDataset(1 to 100).limit(10).filter($"value" % 10 === 0).collect
res1: Array[Int] = Array(10)

How was this patch tested?

Pass the Jenkins with a new test case.

[gatorsmile]

Writing an end-to-end test case is not very helpful here.

You can write a new test case in the related optimizer rule.

[dongjoon-hyun]

Thank you, @gatorsmile .

[SparkQA]

Test build #63548 has finished for PR 14583 at commit aa84c8e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63581 has finished for PR 14583 at commit d23d348.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please

[SparkQA]

Test build #63589 has finished for PR 14583 at commit d23d348.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63732 has finished for PR 14583 at commit 91fb344.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @rxin .
Could you review this PR about PushDownPredicate?

[SparkQA]

Test build #63854 has finished for PR 14583 at commit c09fdc6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63986 has finished for PR 14583 at commit eea0dfe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

I'm fixing this differently here: #14713

[dongjoon-hyun]

Okay. Thanks, @rxin.