[SPARK-19426][SQL] Custom coalesce for Dataset by mariusvniekerk · Pull Request #16766 · apache/spark

mariusvniekerk · 2017-02-01T16:39:56Z

What changes were proposed in this pull request?

This adds support for using the PartitionCoalescer features added in #11865 (SPARK-14042) to the Dataset API

How was this patch tested?

Manual tests

gatorsmile · 2017-02-02T07:33:34Z

Could you follow the styles documented in https://github.com/databricks/scala-style-guide?

gatorsmile · 2017-02-02T07:46:55Z

CoalesceLogical -> Coalesce?

Main reason is there was already a Coalesce expression class

gatorsmile · 2017-02-02T07:48:49Z

Could you please also add a few test cases? For example, DataFrameSuite or DatasetSuite.

felixcheung · 2017-02-02T17:41:11Z

I'd second that. I'd be interested to know if this implementation changes behavior for coalesce

gatorsmile · 2017-02-03T06:56:49Z

Please also add the function description, like what we did in the other functions in Dataset.scala?

gatorsmile · 2017-02-03T06:57:21Z

The same indent issue here.

case class CoalesceExec( numPartitions: Int, child: SparkPlan, partitionCoalescer: Option[PartitionCoalescer]) extends UnaryExecNode {

Do you guys have a .scalafmt.conf that applies all of this? that should make things cleaner.

gatorsmile · 2017-02-03T07:05:39Z

gatorsmile · 2017-02-03T07:12:05Z

The name still looks inconsistent with the others. How about PartitionCoalesce?

that sounds good

gatorsmile · 2017-02-03T07:12:24Z

This description is not right.

gatorsmile · 2017-02-03T07:12:44Z

Style issues:
caseclassPartitionCoalesce( numPartitions: Int, partitionCoalescer: Option[PartitionCoalescer], child: LogicalPlan) extendsUnaryNode {

gatorsmile · 2017-02-03T07:14:22Z

Add it back?

gatorsmile · 2017-02-03T07:19:33Z

cannot resolve '`a`' given input columns: [_c0, _c1];

Oh right csv doesn't do headers.

gatorsmile · 2017-02-03T07:23:48Z

use WithTempPath to generate the path?

gatorsmile · 2017-02-03T07:25:44Z

No need to do it, if you use withTempPath. This is an example

ah thanks. I looked at the writer tests

gatorsmile · 2017-02-03T07:26:14Z

Nit: Remove this empty line.

mariusvniekerk · 2017-02-03T16:23:50Z

@felixcheung This does not touch any of the coalesce internals. Only allows setting a partitionCoalescer similar to what is already available in rdd.coalesce

gatorsmile · 2017-06-13T06:23:19Z

Sorry for the late reply. @mariusvniekerk

Could you please update the PR?

gatorsmile · 2017-06-13T06:23:28Z

ok to test

SparkQA · 2017-06-13T06:29:11Z

Test build #77969 has finished for PR 16766 at commit d4bde0b.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

mariusvniekerk · 2017-06-13T23:14:05Z

Let me rebase this. I don't currently have a clean way of testing this on Windows

gatorsmile · 2017-06-14T06:43:25Z

Could you run the following four commands to do a local test in your local environment?

dev/lint-scala
build/sbt -Phive hive/test
build/sbt sql/test
build/sbt catalyst/test

SparkQA · 2017-06-17T15:19:11Z

Test build #78212 has finished for PR 16766 at commit 00b2a7a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CoalesceExec(

SparkQA · 2017-06-17T15:48:47Z

Test build #78213 has finished for PR 16766 at commit e99cebc.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-17T22:09:27Z

Test build #78218 has finished for PR 16766 at commit 1d1b2fa.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class DataSetSizeBasedPartitionCoalescer(maxSize: Int) extends

HyukjinKwon

Actual javadoc errors are as below:

[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/target/java/org/apache/spark/sql/Dataset.java:2222: error: reference not found
[error] * A {@link PartitionCoalescer} can also be supplied allowing the behavior of the partitioning to be
[error] ^
[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/target/java/org/apache/spark/sql/Dataset.java:2223: error: reference not found
[error] * customized similar to {@link RDD.coalesce}.
[error]

HyukjinKwon · 2017-06-19T04:31:10Z

 * the current partitioning is).
 *
+ * A [[PartitionCoalescer]] can also be supplied allowing the behavior of the partitioning to be
+ * customized similar to [[RDD.coalesce]].


I think it should be [[org.apache.spark.rdd.RDD##coalesce]].

HyukjinKwon · 2017-06-19T04:32:08Z

 * current upstream partitions will be executed in parallel (per whatever
 * the current partitioning is).
 *
+ * A [[PartitionCoalescer]] can also be supplied allowing the behavior of the partitioning to be


Sounds this trait is unable to be generated as is in Java. Simply wrapping `...` would be fine.

HyukjinKwon · 2017-07-24T03:22:55Z

Hi @mariusvniekerk, would you be able to fix the javadoc errors?

gatorsmile · 2017-08-06T06:12:41Z

cc @maropu Do you want to take this over?

maropu · 2017-08-06T07:52:47Z

@gatorsmile Sure! I'll do, Thanks!

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closesapache#18585 - SPARK-21359 Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closesapache#18667 - Fix the simpleString used in error messages Closesapache#18782 - Branch 2.1 Added: Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closesapache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18780 from HyukjinKwon/close-prs.

gatorsmile reviewed Feb 2, 2017
View reviewed changes

gatorsmile reviewed Feb 3, 2017
View reviewed changes

Comment threadsql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala Outdated
gatorsmileFeb 3, 2017
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add it back?

gatorsmile reviewed Feb 3, 2017
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala Outdated
gatorsmileFeb 3, 2017
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Remove this empty line.

mariusvniekerk added 4 commits June 17, 2017 11:12

custom coalesce
0f1060c

Adapted unit test for custom coalesce from the one in RDDSuite
2742457

Addressing review comments.
2499364

Indentation fixup
00b2a7a

mariusvniekerk force-pushed the wip_customCoalesce branch from d4bde0b to 00b2a7aCompare June 17, 2017 15:15

Fixed style issues
e99cebc

Fixed failing tests
1d1b2fa

HyukjinKwon reviewed Jun 19, 2017
View reviewed changes

HyukjinKwon mentioned this pull request Jul 31, 2017
[INFRA] Close stale PRs #18780
Closed

asfgit closed this in 3a45c7fAug 5, 2017

maropu mentioned this pull request Aug 6, 2017
[SPARK-19426][SQL] Custom coalescer for Dataset #18861
Closed

Uh oh!

Conversation

mariusvniekerk commented Feb 1, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Feb 2, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixcheung commented Feb 2, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmileFeb 3, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmileFeb 3, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mariusvniekerk commented Feb 3, 2017

Uh oh!

gatorsmile commented Jun 13, 2017

Uh oh!

gatorsmile commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

mariusvniekerk commented Jun 13, 2017

Uh oh!

gatorsmile commented Jun 14, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jul 24, 2017

Uh oh!

gatorsmile commented Feb 2, 2017•
edited
Loading

gatorsmileFeb 3, 2017•
edited
Loading

gatorsmileFeb 3, 2017•
edited
Loading