Skip to content

[SPARK-26466][CORE] Use ConfigEntry for hardcoded configs for submit categories. - #23532

Closed
HeartSaVioR wants to merge 3 commits into
apache:masterfrom
HeartSaVioR:SPARK-26466-v2
Closed

[SPARK-26466][CORE] Use ConfigEntry for hardcoded configs for submit categories.#23532
HeartSaVioR wants to merge 3 commits into
apache:masterfrom
HeartSaVioR:SPARK-26466-v2

Conversation

@HeartSaVioR

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

The PR makes hardcoded configs below to use ConfigEntry.

  • spark.kryo
  • spark.kryoserializer
  • spark.serializer
  • spark.jars
  • spark.files
  • spark.submit
  • spark.deploy
  • spark.worker

This patch doesn't change configs which are not relevant to SparkConf (e.g. system properties).

How was this patch tested?

Existing tests.

@SparkQA

Copy link
Copy Markdown

Test build #101151 has finished for PR 23532 at commit e76866b.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HeartSaVioR

Copy link
Copy Markdown
ContributorAuthor

Ran compilation in my machine and passed.

mvn -Phadoop-2.7 -Pkubernetes -Pkubernetes-integration-tests -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos clean compile test-compile

@SparkQA

Copy link
Copy Markdown

Test build #101152 has finished for PR 23532 at commit e56b307.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to set this to false because by default it doesn't allow setting configuration which is defined in SparkConf. I'm not fully sure which one is right to do: 1) make the change as I just did 2) don't add spark.kryoserializer.buffer and spark.kryoserializer.buffer.max to Kryo.

@SparkQA

Copy link
Copy Markdown

Test build #101174 has finished for PR 23532 at commit c263104.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowensrowen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good @HeartSaVioR ; could you rebase?

Following are addressed:
* spark.kryo
* spark.kryoserializer
* spark.serializer
* spark.jars
* spark.files
* spark.submit
* spark.deploy
* spark.worker
@HeartSaVioR

Copy link
Copy Markdown
ContributorAuthor

@srowen Thanks for reviewing! I've just rebased.

@SparkQA

Copy link
Copy Markdown

Test build #101323 has finished for PR 23532 at commit b00669d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen

Copy link
Copy Markdown
Member

Merged to master

@HeartSaVioR

Copy link
Copy Markdown
ContributorAuthor

Thanks for reviewing and merging, @srowen !

@HeartSaVioR
HeartSaVioR deleted the SPARK-26466-v2 branch January 25, 2019 22:13
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
…categories.
## What changes were proposed in this pull request?
The PR makes hardcoded configs below to use `ConfigEntry`.
* spark.kryo
* spark.kryoserializer
* spark.serializer
* spark.jars
* spark.files
* spark.submit
* spark.deploy
* spark.worker
This patch doesn't change configs which are not relevant to SparkConf (e.g. system properties).
## How was this patch tested?
Existing tests.
Closesapache#23532 from HeartSaVioR/SPARK-26466-v2.
Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@HeartSaVioR@SparkQA@srowen