[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly by pzzs · Pull Request #5972 · apache/spark

pzzs · 2015-05-07T07:04:15Z

SQL

select key from src where 3 in (4, 5);

Before

== Optimized Logical Plan ==
Project [key#12]
Filter 3 INSET (5,4)
MetastoreRelation default, src, None

After

== Optimized Logical Plan ==
LocalRelation [key#228], []

merge lastest spark

AmplabJenkins · 2015-05-07T07:07:09Z

Can one of the admins verify this patch?

AmplabJenkins · 2015-05-07T08:02:12Z

Can one of the admins verify this patch?

scwf · 2015-05-07T09:44:37Z

please add a test for this

pzzs · 2015-05-07T12:21:20Z

@scwf

yhuai · 2015-05-07T15:25:49Z

ok to test

AmplabJenkins · 2015-05-07T15:27:12Z

Merged build triggered.

AmplabJenkins · 2015-05-07T15:27:21Z

Merged build started.

SparkQA · 2015-05-07T15:30:04Z

Test build #32117 has started for PR 5972 at commit 24739bd.

SparkQA · 2015-05-07T17:20:10Z

Test build #32117 has finished for PR 5972 at commit 24739bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-08T01:38:05Z

Test build #32174 has started for PR 5972 at commit abe2bbb.

yhuai · 2015-05-08T01:40:20Z

Why move it?

because it more like a transform, i think it should before ConstantFolding @yhuai

All of them are in the batch of "ConstantFolding". So, we do not really need to move them, right?

Not necessary, but looks ok to me, here first do transform from in -> inset, then in ConstantFolding it will call InSet.eval (not In.eval) which should be more efficient.

SparkQA · 2015-05-08T01:56:46Z

Test build #32173 has finished for PR 5972 at commit fa461a5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T01:56:49Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T01:56:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32173/
Test FAILed.

AmplabJenkins · 2015-05-08T01:57:11Z

Merged build triggered.

AmplabJenkins · 2015-05-08T01:57:20Z

Merged build started.

SparkQA · 2015-05-08T01:59:03Z

Test build #32177 has started for PR 5972 at commit 4c722a2.

SparkQA · 2015-05-08T02:02:50Z

Test build #32174 has finished for PR 5972 at commit abe2bbb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T02:02:54Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T02:02:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32174/
Test FAILed.

SparkQA · 2015-05-08T03:47:28Z

Test build #32177 has finished for PR 5972 at commit 4c722a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T03:47:32Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T03:47:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32177/
Test PASSed.

pzzs · 2015-05-11T09:00:55Z

@marmbrus @yhuai @scwf

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closes#5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request #13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request #12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request #10 from apache/master f61210c [Zhongshuai Pei] Merge pull request #9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request #8 from apache/master 802261c [DoingDone9] Merge pull request #7 from apache/master d00303b [DoingDone9] Merge pull request #6 from apache/master 98b134f [DoingDone9] Merge pull request #5 from apache/master 161cae3 [DoingDone9] Merge pull request #4 from apache/master c87e8b6 [DoingDone9] Merge pull request #3 from apache/master cb1852d [DoingDone9] Merge pull request #2 from apache/master c3f046f [DoingDone9] Merge pull request #1 from apache/master (cherry picked from commit 4b5e1fe) Signed-off-by: Michael Armbrust <michael@databricks.com>

marmbrus · 2015-05-12T02:26:16Z

Thanks, merged to master and 1.4

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closesapache#5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request apache#13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master 802261c [DoingDone9] Merge pull request apache#7 from apache/master d00303b [DoingDone9] Merge pull request apache#6 from apache/master 98b134f [DoingDone9] Merge pull request apache#5 from apache/master 161cae3 [DoingDone9] Merge pull request apache#4 from apache/master c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master cb1852d [DoingDone9] Merge pull request apache#2 from apache/master c3f046f [DoingDone9] Merge pull request apache#1 from apache/master

Replaced `fs.listFiles` with Hadoop-1 friendly `fs.listStatus` method. Author: Hari Shreedharan <hshreedharan@apache.org> Closes#6619 from harishreedharan/evetlog-hadoop-1-fix and squashes the following commits: 6192078 [Hari Shreedharan] [HOTFIX] Fix Hadoop-1 build caused by #5972.

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closesapache#5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request apache#13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master 802261c [DoingDone9] Merge pull request apache#7 from apache/master d00303b [DoingDone9] Merge pull request apache#6 from apache/master 98b134f [DoingDone9] Merge pull request apache#5 from apache/master 161cae3 [DoingDone9] Merge pull request apache#4 from apache/master c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master cb1852d [DoingDone9] Merge pull request apache#2 from apache/master c3f046f [DoingDone9] Merge pull request apache#1 from apache/master

Replaced `fs.listFiles` with Hadoop-1 friendly `fs.listStatus` method. Author: Hari Shreedharan <hshreedharan@apache.org> Closesapache#6619 from harishreedharan/evetlog-hadoop-1-fix and squashes the following commits: 6192078 [Hari Shreedharan] [HOTFIX] Fix Hadoop-1 build caused by apache#5972.

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closesapache#5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request apache#13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master 802261c [DoingDone9] Merge pull request apache#7 from apache/master d00303b [DoingDone9] Merge pull request apache#6 from apache/master 98b134f [DoingDone9] Merge pull request apache#5 from apache/master 161cae3 [DoingDone9] Merge pull request apache#4 from apache/master c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master cb1852d [DoingDone9] Merge pull request apache#2 from apache/master c3f046f [DoingDone9] Merge pull request apache#1 from apache/master

Replaced `fs.listFiles` with Hadoop-1 friendly `fs.listStatus` method. Author: Hari Shreedharan <hshreedharan@apache.org> Closesapache#6619 from harishreedharan/evetlog-hadoop-1-fix and squashes the following commits: 6192078 [Hari Shreedharan] [HOTFIX] Fix Hadoop-1 build caused by apache#5972.

pzzs added 13 commits March 2, 2015 10:11

Merge pull request#1from apache/master
c3f046f
merge lastest spark

Merge pull request#2from apache/master
cb1852d
merge lastest spark

Merge pull request#3from apache/master
c87e8b6
merge lastest spark

Merge pull request#4from apache/master
161cae3
merge lastest spark

Merge pull request#5from apache/master
98b134f
merge lastest spark

Merge pull request#6from apache/master
d00303b
merge lastest spark

Merge pull request#7from apache/master
802261c
merge lastest spark

Merge pull request#8from apache/master
34b1a9a
merge lastest spark

Merge pull request#9from apache/master
f61210c
merge lastest spark

Merge pull request#10from apache/master
f12fa50
merge lastest spark

Merge pull request#12from apache/master
f03fe7f
merge lastest spark

Merge pull request#13from apache/master
14952e2
merge lastest spark

Update Optimizer.scala
2e8f6ca

pzzs closed this May 7, 2015

pzzs reopened this May 7, 2015

pzzs changed the title ~~[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into false directly if not in.~~[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly if all elements of list is LiteralMay 7, 2015

Update Optimizer.scala
36c194e

pzzs added 3 commits May 7, 2015 19:10

Update Optimizer.scala
35ceb7a

Update ConstantFoldingSuite.scala
f4dbf50

Update ConstantFoldingSuite.scala
24739bd

pzzs changed the title ~~[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly if all elements of list is Literal~~[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directlyMay 7, 2015

yhuai reviewed May 8, 2015
View reviewed changes

Update predicates.scala
4c722a2

asfgit closed this in 4b5e1feMay 12, 2015

harishreedharan added a commit to harishreedharan/spark that referenced this pull request Jun 3, 2015
[HOTFIX] Fix Hadoop-1 build caused byapache#5972.
a5e6f6a

harishreedharan added a commit to harishreedharan/spark that referenced this pull request Jun 3, 2015
[HOTFIX] Fix Hadoop-1 build caused byapache#5972.
6192078

Uh oh!

Conversation

pzzs commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

scwf commented May 7, 2015

Uh oh!

pzzs commented May 7, 2015

Uh oh!

yhuai commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

yhuaiMay 8, 2015

Choose a reason for hiding this comment

Uh oh!

pzzsMay 8, 2015

Choose a reason for hiding this comment

Uh oh!

yhuaiMay 8, 2015

Choose a reason for hiding this comment

Uh oh!

scwfMay 8, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

pzzs commented May 11, 2015

Uh oh!

marmbrus commented May 12, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants