[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming by liupc · Pull Request #23845 · apache/spark

liupc · 2019-02-20T11:29:49Z

What changes were proposed in this pull request?

Currently, when enabled streaming dynamic allocation for streaming applications, the maxNumExecutorFailures in ApplicationMaster is still computed with spark.dynamicAllocation.maxExecutors.

Actually, we should consider spark.streaming.dynamicAllocation.maxExecutors instead.

Related codes:

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala

Line 101 in f87153a

privatevalmaxNumExecutorFailures= {

How was this patch tested?

NA

Please review http://spark.apache.org/contributing.html before opening a pull request.

srowen · 2019-02-20T14:50:03Z

For other reviewers -- this is the fix itself.

SparkQA · 2019-02-22T15:02:05Z

Test build #4565 has finished for PR 23845 at commit d5a5107.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

srowen · 2019-02-25T10:46:44Z

@liupc could you have a look at the build failure?

SparkQA · 2019-02-26T15:54:28Z

Test build #4575 has finished for PR 23845 at commit 155f284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

liupc · 2019-03-01T02:19:13Z

retest this please!

liupc · 2019-03-01T03:26:35Z

@srowen UT seems failed with some unrelated tests, can you help to retest this please?

SparkQA · 2019-03-01T17:35:46Z

Test build #4583 has finished for PR 23845 at commit 155f284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

liupc · 2019-03-05T06:59:56Z

retest this please

liupc · 2019-03-06T06:46:25Z

retest this please

liupc · 2019-03-06T06:47:47Z

cc @HyukjinKwon @srowen

SparkQA · 2019-03-06T15:19:05Z

Test build #4598 has started for PR 23845 at commit f825118.

SparkQA · 2019-03-09T18:08:20Z

Test build #4603 has finished for PR 23845 at commit f825118.

This patch fails Java style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-03-11T14:15:20Z

Test build #4608 has finished for PR 23845 at commit f825118.

This patch fails Java style tests.
This patch does not merge cleanly.
This patch adds no public classes.

srowen · 2019-03-11T14:24:22Z

Hm, that's weird. The failure is not directly related, and I thought we fixed this in #23887 Not sure what's going on. dev/lint-java and dev/sbt-checkstyle pass locally for master too, and this isn't failing other PR builds. Maybe some odd cached artifacts on a Jenkins build server? no idea.

SparkQA · 2019-03-11T14:37:20Z

Test build #4609 has started for PR 23845 at commit f825118.

SparkQA · 2019-03-12T18:53:38Z

Test build #4612 has finished for PR 23845 at commit f825118.

This patch fails Java style tests.
This patch does not merge cleanly.
This patch adds no public classes.

vanzin

I'm really not familiar with streaming dynamic allocation, but I assume you can't enable both that and regular dynamic allocation together?

If for some reason that is allowed, then the fix should take the max of both values.

To fix the tests, probably needs a merge with master. Maybe github is confused.

vanzin · 2019-03-13T00:17:05Z

Could you add these to a new Streaming.scala instead? We shouldn't be adding more stuff to package.scala.

@vanzin
If for some reason that is allowed, then the fix should take the max of both values.
This is not allowed to enable both of them. This check is done in the scheduler/ExecutorAllocationManager
spark/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala
Line 211 in 7beb464
thrownewIllegalArgumentException(
I agree to put these configs to Streaming.scala, it's more clear. I will update.

vanzin · 2019-03-13T22:13:21Z

ok to test

SparkQA · 2019-03-13T22:26:39Z

Test build #103459 has finished for PR 23845 at commit 2f46380.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-15T18:12:25Z

Test build #4625 has finished for PR 23845 at commit 2f46380.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-03-15T18:30:27Z

@liupc can you take a look at ...

[error] /home/jenkins/workspace/NewSparkPullRequestBuilder/core/src/main/scala/org/apache/spark/util/Utils.scala:2495: not found: value STREAMING_DYN_ALLOCATION_ENABLED
[error] val streamingDynamicAllocationEnabled = conf.get(STREAMING_DYN_ALLOCATION_ENABLED)
[error]

SparkQA · 2019-03-16T22:43:38Z

Test build #103570 has finished for PR 23845 at commit a46d4d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-03-17T00:45:12Z

merged to master

liupc changed the title ~~[SPARK-26941]Fix incorrect computation of maxNumExecutorFailures for streaming~~[SPARK-26941][YARN]Fix incorrect computation of maxNumExecutorFailures in ApplicationMaster for streamingFeb 20, 2019

srowen requested changes Feb 20, 2019
View reviewed changes

srowen approved these changes Feb 22, 2019
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/internal/config/package.scala Outdated

vanzin reviewed Mar 13, 2019
View reviewed changes

Liupengcheng added 5 commits March 13, 2019 09:30

Fix incorrect computation of maxNumExecutorFailures for streaming
f513285

Replace hardcoded configs with ConfigEntry
1b1e4d0

Refine as commented
ddc5312

fix streaming is not an enclosing class
bd21ba3

Fix ExecutorAllocationManasterSuite
882bd35

liupc force-pushed the Fix-incorrect-maxNumExecutorFailures-for-streaming branch from f825118 to 882bd35Compare March 13, 2019 01:31

Moving streaming configs to Streaming.scala
2f46380

Add missing Streaming.scala
a46d4d3

srowen closed this in cad475dMar 17, 2019

Uh oh!

Conversation

liupc commented Feb 20, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srowenFeb 20, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Feb 22, 2019

Uh oh!

srowen commented Feb 25, 2019

Uh oh!

SparkQA commented Feb 26, 2019

Uh oh!

liupc commented Mar 1, 2019

Uh oh!

liupc commented Mar 1, 2019

Uh oh!

SparkQA commented Mar 1, 2019

Uh oh!

liupc commented Mar 5, 2019

Uh oh!

liupc commented Mar 6, 2019

Uh oh!

liupc commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 9, 2019

Uh oh!

SparkQA commented Mar 11, 2019

Uh oh!

srowen commented Mar 11, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Mar 11, 2019

Uh oh!

SparkQA commented Mar 12, 2019

Uh oh!

vanzin left a comment

Choose a reason for hiding this comment

Uh oh!

vanzinMar 13, 2019

Choose a reason for hiding this comment

Uh oh!

liupcMar 13, 2019

Choose a reason for hiding this comment

Uh oh!

vanzin commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 15, 2019

Uh oh!

srowen commented Mar 15, 2019

Uh oh!

SparkQA commented Mar 16, 2019

Uh oh!

srowen commented Mar 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

srowen commented Mar 11, 2019•
edited
Loading