[SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins by scwf · Pull Request #15240 · apache/spark

scwf · 2016-09-26T11:44:10Z

What changes were proposed in this pull request?

JIRA: https://issues.apache.org/jira/browse/SPARK-17556
Design doc :
https://issues.apache.org/jira/secure/attachment/12830668/executor%20broadcast.pdf

Added two api for RDD to perform executor side broadcast and apply it on sql's broadcast join.

[1]. def broadcast[U: ClassTag](f: Iterator[T] => U): Broadcast[U]

User only need pass a function to describe how to translate all the element of the rdd to the value they want to broadcast

[2]. def broadcast[U: ClassTag](transFunc: TransFunc[T, U]): Broadcast[U]

This is only used in spark sql(spark internal), TransFunc is a interface to describe how to translate all the element of the rdd to a single value.
TransFunc is inherited by BroadcastMode in spark sql.

When construct broadcast, firstly it write blocks to block manager from executor and then create Broadcast from driver(not write blocks)

How was this patch tested?

added unit test and manual test.

SparkQA · 2016-09-26T11:49:04Z

Test build #65907 has finished for PR 15240 at commit 89336e7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T12:18:01Z

Test build #65909 has finished for PR 15240 at commit f0a26c3.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T14:10:49Z

Test build #65913 has finished for PR 15240 at commit 9a334df.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T14:28:30Z

Test build #65916 has finished for PR 15240 at commit 90217c0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T17:02:35Z

Test build #65914 has finished for PR 15240 at commit 5e2ee76.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T17:03:03Z

Test build #65918 has finished for PR 15240 at commit 5db6a3c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T19:07:38Z

Test build #65925 has finished for PR 15240 at commit 4714cae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-28T10:38:58Z

Test build #66030 has finished for PR 15240 at commit 0b80837.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-29T08:40:24Z

Test build #66088 has finished for PR 15240 at commit 3e77821.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-29T11:59:02Z

Test build #66095 has finished for PR 15240 at commit 147b053.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

scwf · 2016-10-08T00:57:41Z

/cc @rxin can you help review this?

SparkQA · 2016-12-30T11:15:49Z

Test build #70745 has finished for PR 15240 at commit cdab885.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-03T07:17:43Z

Test build #70804 has started for PR 15240 at commit cdab885.

SparkQA · 2017-01-03T10:15:48Z

Test build #70807 has finished for PR 15240 at commit 26b03b2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-03T14:44:50Z

Test build #70818 has finished for PR 15240 at commit 26b03b2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

scwf · 2017-01-03T14:49:23Z

retest this please

This reverts commit 76dfc20.

…adcast

SparkQA · 2017-01-04T12:20:41Z

Test build #70867 has finished for PR 15240 at commit 4a547a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2017-06-13T02:57:56Z

Are you still working on this? @scwf

viirya · 2017-06-13T04:41:21Z

@jiangxb1987 there's another solution to this JIRA at #15178. We take different approaches. You can also take a look.

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closesapache#18585 - SPARK-21359 Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closesapache#18667 - Fix the simpleString used in error messages Closesapache#18782 - Branch 2.1 Added: Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closesapache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18780 from HyukjinKwon/close-prs.

scwf force-pushed the SPARK-17556 branch from 4714cae to 0b80837Compare September 28, 2016 08:12

scwf changed the title ~~[SPARK-17556] Executor side broadcast for broadcast joins~~[SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joinsSep 29, 2016

scwfand others added 9 commits December 30, 2016 15:14

executor-side broadcast
0b0a252

renaming better
31eaefb

add finally
9dbdd8c

improve the rdd broadcast api
55c2b8e

rename improvement
8347ad6

var => val
a8c7a54

add shutdown hook to delete dir
41110f4

fix dagschedulersuite
1bb8a44

improve and fix compile error
cdab885

scwf force-pushed the SPARK-17556 branch from 147b053 to cdab885Compare December 30, 2016 09:09

checkSum leads to NPE
26b03b2

clean check sum
76dfc20

scwf added 2 commits January 4, 2017 16:38

Revert "clean check sum"
482c0c8
This reverts commit 76dfc20.

added transient for obj, option leads to obj as a field of TorrentBro…
4a547a6
…adcast

HyukjinKwon mentioned this pull request Jul 31, 2017
[INFRA] Close stale PRs #18780
Closed

asfgit closed this in 3a45c7fAug 5, 2017

Uh oh!

Conversation

scwf commented Sep 26, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 28, 2016

Uh oh!

SparkQA commented Sep 29, 2016

Uh oh!

SparkQA commented Sep 29, 2016

Uh oh!

scwf commented Oct 8, 2016

Uh oh!

SparkQA commented Dec 30, 2016

Uh oh!

SparkQA commented Jan 3, 2017

Uh oh!

SparkQA commented Jan 3, 2017

Uh oh!

SparkQA commented Jan 3, 2017

Uh oh!

scwf commented Jan 3, 2017

Uh oh!

SparkQA commented Jan 4, 2017

Uh oh!

jiangxb1987 commented Jun 13, 2017

Uh oh!

viirya commented Jun 13, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

scwf commented Sep 26, 2016•
edited
Loading

viirya commented Jun 13, 2017•
edited
Loading