[SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting by bdrillard · Pull Request #18354 · apache/spark

bdrillard · 2017-06-19T21:36:20Z

What changes were proposed in this pull request?

This is a backport patch for Spark 2.1.x of the class splitting feature over excess generated code as was merged in #18075.

How was this patch tested?

The same test provided in #18075 is included in this patch.

maor121 · 2017-06-20T08:04:00Z

This is a very important fix for a very common use case. In our company we had to create a workaround in many different cases. Are there any plans to merge this into 2.0.x as well?
We cannot upgrade to 2.1, and we really need this fix on 2.0.x as well :)

bdrillard · 2017-06-20T14:39:43Z

@maor121 Yeah, I should be able to backpatch to 2.0.x as well in a separate PR.

dongjoon-hyun · 2017-06-20T19:52:33Z

Hi, @bdrillard .
Did you make a backport for branch-2.2?

bdrillard · 2017-06-20T19:57:39Z

@dongjoon-hyun I had not yet. I hadn't realized master was already on a 2.3.0-SNAPSHOT. I can also write a backport for branch-2.2.

dongjoon-hyun · 2017-06-20T20:07:50Z

Yep. To backport, we must land this on branch-2.2 first.
In addition, you can use the same title except branch name like the following.

[SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation: Constant Pool Limit - Class Splitting
[SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting

bdrillard · 2017-06-20T20:11:25Z

@dongjoon-hyun Great, thanks for the heads up. I'll prepare a branch-2.2 backport PR and circle back to this PR with the link as confirmation.

bdrillard · 2017-06-21T15:06:21Z

@dongjoon-hyun I've prepared the PR for the branch-2.2 backport, #18377.

cloud-fan · 2017-06-21T16:00:54Z

ok to test

SparkQA · 2017-06-21T17:57:51Z

Test build #78394 has finished for PR 18354 at commit 5685072.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-21T20:40:15Z

Test build #78401 has finished for PR 18354 at commit 51a59cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ol Limit - Class Splitting ## What changes were proposed in this pull request? This is a backport patch for Spark 2.1.x of the class splitting feature over excess generated code as was merged in #18075. ## How was this patch tested? The same test provided in #18075 is included in this patch. Author: ALeksander Eskilson <alek.eskilson@cerner.com> Closes#18354 from bdrillard/class_splitting_2.1.

cloud-fan · 2017-06-22T05:24:39Z

thanks, merging to 2.1!

HyukjinKwon · 2017-07-24T02:48:32Z

@bdrillard, sounds properly backported. Would you close this?

manojlds · 2017-07-24T07:36:55Z

Finding it difficult to track. Is this available in a 2.1 release?

cloud-fan · 2017-07-24T10:17:37Z

it was backported and then reverted because it's too risky, we can close this PR.

manojlds · 2017-07-24T10:30:53Z

@cloud-fan - thanks. Is there any known workaround for this?

cloud-fan · 2017-07-27T07:41:42Z

Unfortunately there is no workaround... You can upgrade to spark 2.3 or do the backport and make a new release in your company. cc @rxin any ideas for this case?

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closesapache#18585 - SPARK-21359 Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closesapache#18667 - Fix the simpleString used in error messages Closesapache#18782 - Branch 2.1 Added: Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closesapache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18780 from HyukjinKwon/close-prs.

ALeksander Eskilson added 2 commits June 19, 2017 16:25

class_splitting_2.1 creating backpatch of class_splitting_only
52b654a

class_splitting_2.1 adding doc for class_splitting functions
e2f8fd2

bdrillard changed the title ~~Class splitting 2.1~~[SPARK-18016][SQL][Catalyst] Backport of Class Splitting, #18075Jun 19, 2017

bdrillard changed the title ~~[SPARK-18016][SQL][Catalyst] Backport of Class Splitting, #18075~~[SPARK-18016][SQL][Catalyst] Backport of Class SplittingJun 19, 2017

class_splitting_only_2.1 fixing doc and whitespace
5685072

bdrillard changed the title ~~[SPARK-18016][SQL][Catalyst] Backport of Class Splitting~~[SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class SplittingJun 20, 2017

class_splitting_2.1 fixing test rows
51a59cb

bdrillard mentioned this pull request Jul 24, 2017
[SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation: Constant Pool Limit - Class Splitting #18377
Closed

HyukjinKwon mentioned this pull request Jul 31, 2017
[INFRA] Close stale PRs #18780
Closed

asfgit closed this in 3a45c7fAug 5, 2017

Uh oh!

Conversation

bdrillard commented Jun 19, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

maor121 commented Jun 20, 2017

Uh oh!

bdrillard commented Jun 20, 2017

Uh oh!

dongjoon-hyun commented Jun 20, 2017

Uh oh!

bdrillard commented Jun 20, 2017

Uh oh!

dongjoon-hyun commented Jun 20, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bdrillard commented Jun 20, 2017

Uh oh!

bdrillard commented Jun 21, 2017

Uh oh!

cloud-fan commented Jun 21, 2017

Uh oh!

SparkQA commented Jun 21, 2017

Uh oh!

SparkQA commented Jun 21, 2017

Uh oh!

cloud-fan commented Jun 22, 2017

Uh oh!

HyukjinKwon commented Jul 24, 2017

Uh oh!

manojlds commented Jul 24, 2017

Uh oh!

cloud-fan commented Jul 24, 2017

Uh oh!

manojlds commented Jul 24, 2017

Uh oh!

cloud-fan commented Jul 27, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dongjoon-hyun commented Jun 20, 2017•
edited
Loading