[SPARK-1495][SQL]add support for left semi join by adrian-wang · Pull Request #837 · apache/spark

adrian-wang · 2014-05-20T09:35:12Z

Just submit another solution for #395

adrian-wang · 2014-05-20T09:37:10Z

This is a solution with #418 from @marmbrus .

AmplabJenkins · 2014-05-20T09:37:58Z

Can one of the admins verify this patch?

rxin · 2014-05-21T08:27:48Z

Jenkins, add to whitelist.

AmplabJenkins · 2014-05-21T08:28:00Z

Merged build triggered.

AmplabJenkins · 2014-05-21T08:28:09Z

Merged build started.

AmplabJenkins · 2014-05-21T08:30:04Z

Merged build finished.

AmplabJenkins · 2014-05-21T08:30:04Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15115/

AmplabJenkins · 2014-05-21T08:47:57Z

Merged build triggered.

AmplabJenkins · 2014-05-21T08:48:04Z

Merged build started.

AmplabJenkins · 2014-05-21T10:19:08Z

Merged build finished.

AmplabJenkins · 2014-05-21T10:19:08Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15116/

AmplabJenkins · 2014-05-21T15:52:58Z

Merged build triggered.

AmplabJenkins · 2014-05-21T15:53:06Z

Merged build started.

AmplabJenkins · 2014-05-21T17:24:14Z

Merged build finished.

AmplabJenkins · 2014-05-21T17:24:14Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15121/

marmbrus · 2014-05-31T18:34:29Z

Intent only 4 spaces here.

marmbrus · 2014-05-31T18:55:51Z

This is getting closer. Thanks for working on it!

marmbrus · 2014-05-31T18:56:57Z

Can you add "[SPARK-1495][SQL]" to the PR title?

AmplabJenkins · 2014-06-03T05:57:58Z

Merged build triggered.

adrian-wang · 2014-06-03T06:01:40Z

Hi Michael, when I was adding the Scala doc, I realized that if the join is not calculated in LeftSemiJoinHash, then it simply means there's no join keys for the left semi join. Then if I pushed down those predicates and conditions(all of them can be pushed down), I only need to verify the if right table size is null here, to decide whether to output the left table. So LeftSemiJoinBNL would be very much excessive. Am I right?

AmplabJenkins · 2014-06-03T07:12:46Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-03T07:12:46Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15374/

marmbrus · 2014-06-07T17:08:04Z

I think in general we should avoid making too many assumptions in the planner about what optimizations have occurred. For example, in the future we might avoid pushing down predicates that are very expensive to evaluate as it might be cheaper to run them on an already filtered set of data. However, in the case of LEFT SEMI JOIN, I think it is actually okay to push all evaluation into the join condition, even if they only refer to the left table. Is that true?

Yes, I think LEFT SEMI JOIN would not suffer by pushing down predicates.

marmbrus · 2014-06-07T19:24:28Z

(I deleted my earlier comment because I found a mistake)

I think this is looking pretty good, but we should at least add one test for the Broadcast Nested Loop version. Here's a PR against your branch that does that: adrian-wang#1

Add a test.

AmplabJenkins · 2014-06-08T06:37:50Z

Merged build triggered.

AmplabJenkins · 2014-06-08T06:53:51Z

Merged build started.

AmplabJenkins · 2014-06-08T08:09:10Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-08T08:09:10Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15540/

Just submit another solution for #395 Author: Daoyuan <daoyuan.wang@intel.com> Author: Michael Armbrust <michael@databricks.com> Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes#837 from adrian-wang/left-semi-join-support and squashes the following commits: d39cd12 [Daoyuan Wang] Merge pull request #1 from marmbrus/pr/837 6713c09 [Michael Armbrust] Better debugging for failed query tests. 035b73e [Michael Armbrust] Add test for left semi that can't be done with a hash join. 5ec6fa4 [Michael Armbrust] Add left semi to SQL Parser. 4c726e5 [Daoyuan] improvement according to Michael 8d4a121 [Daoyuan] add golden files for leftsemijoin 83a3c8a [Daoyuan] scala style fix 14cff80 [Daoyuan] add support for left semi join (cherry picked from commit 0cf6002) Signed-off-by: Michael Armbrust <michael@databricks.com>

marmbrus · 2014-06-09T18:34:59Z

Thanks! I've merged this into 1.0 and master.

Some improvement for PR #837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closes#1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com>

Some improvement for PR #837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closes#1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com> (cherry picked from commit ce6deb1) Signed-off-by: Michael Armbrust <michael@databricks.com>

Just submit another solution for apache#395 Author: Daoyuan <daoyuan.wang@intel.com> Author: Michael Armbrust <michael@databricks.com> Author: Daoyuan Wang <daoyuan.wang@intel.com> Closesapache#837 from adrian-wang/left-semi-join-support and squashes the following commits: d39cd12 [Daoyuan Wang] Merge pull request apache#1 from marmbrus/pr/837 6713c09 [Michael Armbrust] Better debugging for failed query tests. 035b73e [Michael Armbrust] Add test for left semi that can't be done with a hash join. 5ec6fa4 [Michael Armbrust] Add left semi to SQL Parser. 4c726e5 [Daoyuan] improvement according to Michael 8d4a121 [Daoyuan] add golden files for leftsemijoin 83a3c8a [Daoyuan] scala style fix 14cff80 [Daoyuan] add support for left semi join

Some improvement for PR apache#837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closesapache#1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com>

Just submit another solution for apache#395 Author: Daoyuan <daoyuan.wang@intel.com> Author: Michael Armbrust <michael@databricks.com> Author: Daoyuan Wang <daoyuan.wang@intel.com> Closesapache#837 from adrian-wang/left-semi-join-support and squashes the following commits: d39cd12 [Daoyuan Wang] Merge pull request apache#1 from marmbrus/pr/837 6713c09 [Michael Armbrust] Better debugging for failed query tests. 035b73e [Michael Armbrust] Add test for left semi that can't be done with a hash join. 5ec6fa4 [Michael Armbrust] Add left semi to SQL Parser. 4c726e5 [Daoyuan] improvement according to Michael 8d4a121 [Daoyuan] add golden files for leftsemijoin 83a3c8a [Daoyuan] scala style fix 14cff80 [Daoyuan] add support for left semi join

Some improvement for PR apache#837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closesapache#1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com>

…tastore version (apache#837)

add support for left semi join
14cff80

adrian-wang mentioned this pull request May 21, 2014
support leftsemijoin for sparkSQL #395
Closed

scala style fix
83a3c8a

add golden files for leftsemijoin
8d4a121

marmbrus reviewed May 31, 2014
View reviewed changes

Comment threadsql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala Outdated
marmbrusMay 31, 2014
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Intent only 4 spaces here.

adrian-wang changed the title ~~add support for left semi join~~[SPARK-1495][SQL]add support for left semi joinJun 2, 2014

improvement according to Michael
4c726e5

marmbrus reviewed Jun 7, 2014
View reviewed changes

marmbrus added 3 commits June 7, 2014 12:22

Add left semi to SQL Parser.
5ec6fa4

Add test for left semi that can't be done with a hash join.
035b73e

Better debugging for failed query tests.
6713c09

Merge pull request#1from marmbrus/pr/837
d39cd12
Add a test.

asfgit closed this in 0cf6002Jun 9, 2014

adrian-wang mentioned this pull request Jun 11, 2014
Clean left semi join hash #1049
Closed

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022
MapR [SPARK-891] Thrift server start fails due to unsupported Hive Me…
ac5e16b
…tastore version (apache#837)

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024
MapR [SPARK-891] Thrift server start fails due to unsupported Hive Me…
6f189e1
…tastore version (apache#837)

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025
MapR [SPARK-891] Thrift server start fails due to unsupported Hive Me…
3642fa3
…tastore version (apache#837)

Uh oh!

Conversation

adrian-wang commented May 20, 2014

Uh oh!

adrian-wang commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

rxin commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

AmplabJenkins commented May 21, 2014

Uh oh!

marmbrusMay 31, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented May 31, 2014

Uh oh!

marmbrus commented May 31, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

adrian-wang commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

marmbrusJun 7, 2014

Choose a reason for hiding this comment

Uh oh!

adrian-wangJun 8, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Jun 7, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

AmplabJenkins commented Jun 8, 2014

Uh oh!

marmbrus commented Jun 9, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants