[SPARK-15616] [SQL] Metastore relation should fallback to HDFS size of partitions that are involved in Query for JoinSelection. by lianhuiwang · Pull Request #13373 · apache/spark

lianhuiwang · 2016-05-28T05:41:40Z

What changes were proposed in this pull request?

Currently if some partitions of a partitioned table are used in join operation we rely on Metastore returned size of table to calculate if we can convert the operation to Broadcast join.
if Filter can prune some partitions, Hive can prune partition before determining to use broadcast joins according to HDFS size of partitions that are involved in Query.So sparkSQL needs it that can improve join's performance for partitioned table.

How was this patch tested?

integration tests

SparkQA · 2016-05-28T05:43:47Z

Test build #59552 has finished for PR 13373 at commit 77737f1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-28T09:00:29Z

Test build #59555 has finished for PR 13373 at commit ca78723.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-06T02:23:26Z

Test build #60017 has finished for PR 13373 at commit dd6bdf0.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class PushFilterIntoRelation(conf: SQLConf) extends Rule[LogicalPlan] with PredicateHelper
- case class PushProjectIntoRelation(conf: SQLConf) extends Rule[LogicalPlan]

SparkQA · 2016-06-06T03:04:11Z

Test build #60018 has finished for PR 13373 at commit 8b9b07d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-06T13:49:59Z

Test build #60044 has finished for PR 13373 at commit ab86f14.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-07T03:36:05Z

Test build #60094 has finished for PR 13373 at commit fc25f72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-07T08:01:51Z

Test build #60110 has finished for PR 13373 at commit c4f9bc6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-22T03:48:58Z

Test build #62707 has finished for PR 13373 at commit f3da998.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-22T03:55:20Z

Test build #62710 has finished for PR 13373 at commit 2d9e321.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-22T04:04:24Z

Test build #62711 has finished for PR 13373 at commit 1e0a6f2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

lianhuiwang · 2016-07-22T04:57:03Z

cc @cloud-fan @rxin @hvanhovell

SparkQA · 2016-07-22T05:26:35Z

Test build #62704 has finished for PR 13373 at commit 4a3e72e.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2016-07-22T06:43:42Z

Test build #62713 has finished for PR 13373 at commit b573919.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-22T06:57:36Z

Test build #62715 has finished for PR 13373 at commit bf74b0e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-11T15:02:29Z

Test build #63612 has finished for PR 13373 at commit c7b181e.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2016-08-11T17:13:38Z

Test build #63614 has finished for PR 13373 at commit 7d5371a.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

…on_broadcast # Conflicts: # sql/hive/src/main/scala/org/apache/spark/sql/hive/MetastoreRelation.scala

SparkQA · 2016-10-29T15:18:55Z

Test build #67765 has finished for PR 13373 at commit 76a847c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-29T15:48:35Z

Test build #67768 has finished for PR 13373 at commit 5d90974.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-29T18:55:30Z

Test build #67771 has finished for PR 13373 at commit 1a63649.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-30T05:13:22Z

Test build #67777 has finished for PR 13373 at commit a6e2c57.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-06-02T14:14:34Z

@lianhuiwang, I understand it is painful to keep the PR up-to-date. However, shouldn't we probably have the Jenkins build passed at the last even if it has conflicts?

cloud-fan · 2017-06-02T17:01:28Z

Sorry that I think it's not valid anymore after we have PruneFileSourcePartitions rule.

lianhuiwang · 2017-06-03T09:07:01Z

@cloud-fan I do not think that PruneFileSourcePartitions rule is for Hive's CatalogRelation. example in this PR with master branch cannot get expected result. So i will update it with the latest code.

lianhuiwang · 2017-06-04T07:05:29Z

@HyukjinKwon @cloud-fan I will close this PR and create new PR #18193 for it. Thanks.

lianhuiwang added 3 commits May 28, 2016 10:04

init commit
0570a1a

Merge branch 'apache-master' into partition_broadcast
afbe937

update
77737f1

fix style
ca78723

update
dd6bdf0

fix errors
8b9b07d

fix unit test
ab86f14

update
fc25f72

lianhuiwang added 2 commits June 7, 2016 12:11

Merge branch 'apache-master' into partition_broadcast
c0fc69e

fix unit test
c4f9bc6

lianhuiwang mentioned this pull request Jun 23, 2016
[SPARK-15752] [SQL] Optimize metadata only query that has an aggregate whose children are deterministic project or filter operators. #13494
Closed

lianhuiwang added 2 commits July 22, 2016 11:16

update
4a3e72e

update HiveStrategies
1999eb4

Merge branch 'apache-master' into partition_broadcast
f3da998

update with master
2d9e321

update name
1e0a6f2

add ut
b573919

refactor ut
bf74b0e

update
c7b181e

update
7d5371a

Merge branch 'master' ofhttps://github.com/apache/sparkinto partiti…
76a847c
…on_broadcast # Conflicts: # sql/hive/src/main/scala/org/apache/spark/sql/hive/MetastoreRelation.scala

fix style
5d90974

fix some bugs
1a63649

fix some uts
a6e2c57

lianhuiwang closed this Jun 4, 2017

Uh oh!

Conversation

lianhuiwang commented May 28, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 28, 2016

Uh oh!

SparkQA commented May 28, 2016

Uh oh!

SparkQA commented Jun 6, 2016

Uh oh!

SparkQA commented Jun 6, 2016

Uh oh!

SparkQA commented Jun 6, 2016

Uh oh!

SparkQA commented Jun 7, 2016

Uh oh!

SparkQA commented Jun 7, 2016

Uh oh!

SparkQA commented Jul 22, 2016

Uh oh!

SparkQA commented Jul 22, 2016

Uh oh!

SparkQA commented Jul 22, 2016

Uh oh!

lianhuiwang commented Jul 22, 2016

Uh oh!

SparkQA commented Jul 22, 2016

Uh oh!

SparkQA commented Jul 22, 2016

Uh oh!

SparkQA commented Jul 22, 2016

Uh oh!

SparkQA commented Aug 11, 2016

Uh oh!

SparkQA commented Aug 11, 2016

Uh oh!

SparkQA commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 30, 2016

Uh oh!

HyukjinKwon commented Jun 2, 2017

Uh oh!

cloud-fan commented Jun 2, 2017

Uh oh!

lianhuiwang commented Jun 3, 2017

Uh oh!

lianhuiwang commented Jun 4, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants