Skip to content

[SPARK-7267][SQL]Push down Project when it's child is Limit - #5797

Closed
pzzs wants to merge 14 commits into
apache:masterfrom
pzzs:ProjectLimit
Closed

[SPARK-7267][SQL]Push down Project when it's child is Limit#5797
pzzs wants to merge 14 commits into
apache:masterfrom
pzzs:ProjectLimit

Conversation

@pzzs

@pzzspzzs commented Apr 30, 2015

Copy link
Copy Markdown
Contributor

SQL

select key from (select key,value from t1 limit 100) t2 limit 10

Optimized Logical Plan before modifying

== Optimized Logical Plan ==
Limit 10
Project key#228
Limit 100
MetastoreRelation default, t1, None

Optimized Logical Plan after modifying

== Optimized Logical Plan ==
Limit 10
Limit 100
Project key#228
MetastoreRelation default, t1, None

After this, we can combine limits

@rxin

rxin commented Apr 30, 2015

Copy link
Copy Markdown
Contributor

Jenkins, test this please.

@SparkQA

Copy link
Copy Markdown

Test build #31367 has finished for PR 5797 at commit dc83ae9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@scwf

scwf commented Apr 30, 2015

Copy link
Copy Markdown
Contributor

Retest this please

@rxin

rxin commented Apr 30, 2015

Copy link
Copy Markdown
Contributor

Jenkins, test this please.

@SparkQA

Copy link
Copy Markdown

Test build #31396 has finished for PR 5797 at commit 70d0fca.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch adds the following new dependencies:
    • jaxb-api-2.2.7.jar
    • jaxb-core-2.2.7.jar
    • jaxb-impl-2.2.7.jar
    • pmml-agent-1.1.15.jar
    • pmml-model-1.1.15.jar
    • pmml-schema-1.1.15.jar
  • This patch removes the following dependencies:
    • activation-1.1.jar
    • jaxb-api-2.2.2.jar
    • jaxb-impl-2.2.3-1.jar

@rxin

rxin commented Apr 30, 2015

Copy link
Copy Markdown
Contributor

Thanks. Merging in master.

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
SQL
```
select key from (select key,value from t1 limit 100) t2 limit 10
```
Optimized Logical Plan before modifying
```
== Optimized Logical Plan ==
Limit 10
Project key#228
Limit 100
MetastoreRelation default, t1, None
```
Optimized Logical Plan after modifying
```
== Optimized Logical Plan ==
Limit 10
Limit 100
Project key#228
MetastoreRelation default, t1, None
```
After this, we can combine limits
Author: Zhongshuai Pei <799203320@qq.com>
Author: DoingDone9 <799203320@qq.com>
Closesapache#5797 from DoingDone9/ProjectLimit and squashes the following commits:
70d0fca [Zhongshuai Pei] Update FilterPushdownSuite.scala
dc83ae9 [Zhongshuai Pei] Update FilterPushdownSuite.scala
485c61c [Zhongshuai Pei] Update Optimizer.scala
f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master
f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master
f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master
34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master
802261c [DoingDone9] Merge pull request apache#7 from apache/master
d00303b [DoingDone9] Merge pull request apache#6 from apache/master
98b134f [DoingDone9] Merge pull request apache#5 from apache/master
161cae3 [DoingDone9] Merge pull request apache#4 from apache/master
c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master
cb1852d [DoingDone9] Merge pull request apache#2 from apache/master
c3f046f [DoingDone9] Merge pull request apache#1 from apache/master
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
SQL
```
select key from (select key,value from t1 limit 100) t2 limit 10
```
Optimized Logical Plan before modifying
```
== Optimized Logical Plan ==
Limit 10
Project key#228
Limit 100
MetastoreRelation default, t1, None
```
Optimized Logical Plan after modifying
```
== Optimized Logical Plan ==
Limit 10
Limit 100
Project key#228
MetastoreRelation default, t1, None
```
After this, we can combine limits
Author: Zhongshuai Pei <799203320@qq.com>
Author: DoingDone9 <799203320@qq.com>
Closesapache#5797 from DoingDone9/ProjectLimit and squashes the following commits:
70d0fca [Zhongshuai Pei] Update FilterPushdownSuite.scala
dc83ae9 [Zhongshuai Pei] Update FilterPushdownSuite.scala
485c61c [Zhongshuai Pei] Update Optimizer.scala
f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master
f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master
f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master
34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master
802261c [DoingDone9] Merge pull request apache#7 from apache/master
d00303b [DoingDone9] Merge pull request apache#6 from apache/master
98b134f [DoingDone9] Merge pull request apache#5 from apache/master
161cae3 [DoingDone9] Merge pull request apache#4 from apache/master
c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master
cb1852d [DoingDone9] Merge pull request apache#2 from apache/master
c3f046f [DoingDone9] Merge pull request apache#1 from apache/master
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
SQL
```
select key from (select key,value from t1 limit 100) t2 limit 10
```
Optimized Logical Plan before modifying
```
== Optimized Logical Plan ==
Limit 10
Project key#228
Limit 100
MetastoreRelation default, t1, None
```
Optimized Logical Plan after modifying
```
== Optimized Logical Plan ==
Limit 10
Limit 100
Project key#228
MetastoreRelation default, t1, None
```
After this, we can combine limits
Author: Zhongshuai Pei <799203320@qq.com>
Author: DoingDone9 <799203320@qq.com>
Closesapache#5797 from DoingDone9/ProjectLimit and squashes the following commits:
70d0fca [Zhongshuai Pei] Update FilterPushdownSuite.scala
dc83ae9 [Zhongshuai Pei] Update FilterPushdownSuite.scala
485c61c [Zhongshuai Pei] Update Optimizer.scala
f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master
f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master
f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master
34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master
802261c [DoingDone9] Merge pull request apache#7 from apache/master
d00303b [DoingDone9] Merge pull request apache#6 from apache/master
98b134f [DoingDone9] Merge pull request apache#5 from apache/master
161cae3 [DoingDone9] Merge pull request apache#4 from apache/master
c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master
cb1852d [DoingDone9] Merge pull request apache#2 from apache/master
c3f046f [DoingDone9] Merge pull request apache#1 from apache/master
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pzzs@rxin@SparkQA@scwf