[SPARK-32643][CORE][K8s] Consolidate state decommissioning in the TaskSchedulerImpl realm by agrawaldevesh · Pull Request #29452 · apache/spark

agrawaldevesh · 2020-08-18T00:34:50Z

What changes were proposed in this pull request?

The decommissioning state is a bit fragment across two places in the TaskSchedulerImpl:

#29014 stored the incoming decommission info messages in TaskSchedulerImpl.executorsPendingDecommission.
While #28619 was storing just the executor end time in the map TaskSetManager.tidToExecutorKillTimeMapping (which in turn is contained in TaskSchedulerImpl).
While the two states are not really overlapping, it's a bit of a code hygiene concern to save this state in two places.

With #29422, TaskSchedulerImpl is emerging as the place where all decommissioning book keeping is kept within the driver. So consolidate the information in tidToExecutorKillTimeMapping into executorsPendingDecommission.

However, in order to do so, we need to walk away from keeping the raw ExecutorDecommissionInfo messages and instead keep another class ExecutorDecommissionState. This decoupling will allow the RPC message class ExecutorDecommissionInfo to evolve independently from the book keeping ExecutorDecommissionState.

Why are the changes needed?

This is just a code cleanup. These two features were added independently and its time to consolidate their state for good hygiene.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests.

agrawaldevesh · 2020-08-18T03:09:16Z

cc: @holdenk and @prakharjain09 ... This PR simply does some state cleanup/consolidation without making any semantic changes. I would be grateful for your review. I have also created a Jira associated with this. Thanks !

Also @Ngone51 and @cloud-fan

SparkQA · 2020-08-18T03:09:35Z

Test build #127518 has finished for PR 29452 at commit 61ac7b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExecutorDecommissionState(message: String,

Ngone51 · 2020-08-18T03:27:34Z

@agrawaldevesh Could you please add the [CORE] tag in the PR title like other PRs?

SparkQA · 2020-08-18T06:04:31Z

Test build #127522 has finished for PR 29452 at commit 97ad7fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExecutorDecommissionState(message: String,

SparkQA · 2020-08-18T18:49:39Z

Test build #127590 has finished for PR 29452 at commit 7449fa2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ExecutorDecommissionState(

SparkQA · 2020-08-19T06:34:23Z

Test build #127625 has finished for PR 29452 at commit 6a5be83.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-19T07:05:02Z

Test build #127627 has finished for PR 29452 at commit 9222f05.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-08-19T07:11:20Z

retest this please

SparkQA · 2020-08-19T10:17:24Z

Test build #127629 has finished for PR 29452 at commit 9222f05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-19T19:24:18Z

Test build #127654 has finished for PR 29452 at commit ebd8408.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2020-08-19T19:26:39Z

Can we tag this PR with Kubernetes (add [K8S] to the title) so it runs the K8s integration tests?

SparkQA · 2020-08-19T23:04:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32287/

SparkQA · 2020-08-19T23:08:48Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32287/

SparkQA · 2020-08-25T23:21:39Z

Test build #127895 has finished for PR 29452 at commit 9389ed5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-26T03:04:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32531/

SparkQA · 2020-08-26T03:22:57Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32531/

SparkQA · 2020-08-26T04:43:54Z

Test build #127905 has finished for PR 29452 at commit 6b9ded4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

agrawaldevesh · 2020-08-26T05:19:21Z

retest this please

agrawaldevesh · 2020-08-26T05:21:26Z

jenkins retest this please

SparkQA · 2020-08-26T06:06:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32539/

SparkQA · 2020-08-26T06:32:07Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32539/

SparkQA · 2020-08-26T07:05:02Z

Test build #127913 has finished for PR 29452 at commit 6b9ded4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-26T10:04:28Z

Test build #127917 has finished for PR 29452 at commit 29fe131.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

agrawaldevesh · 2020-08-26T17:05:10Z

Hi @holdenk, This PR is back in your court to review again. @Ngone51 is fine with the new set of changes. If it looks good, please feel free to merge it in. Thank you !

holdenk · 2020-08-26T18:45:37Z

Jenkins retest this please

SparkQA · 2020-08-26T19:29:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32559/

SparkQA · 2020-08-26T19:48:54Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32559/

holdenk · 2020-08-26T19:57:06Z

This looks good to me I'm going to go ahead and merge this.

SparkQA · 2020-08-26T21:47:32Z

Test build #127933 has finished for PR 29452 at commit 29fe131.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

agrawaldevesh · 2020-08-27T15:54:07Z

Thank you @holdenk for shepherding this all the way through !

holdenk · 2020-08-27T16:01:21Z

Thanks for working on this so much! I’m really excited to launch this feature in 3.1 👍

probot-autolabelerBot added the CORE label Aug 18, 2020

agrawaldevesh force-pushed the consolidate_decom_state branch from 61ac7b8 to 97ad7feCompare August 18, 2020 03:08

agrawaldevesh changed the title ~~[SPARK-32643] Consolidate state decommissioning in the TaskSchedulerImpl realm~~[SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realmAug 18, 2020

cloud-fan reviewed Aug 18, 2020
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/scheduler/ExecutorDecommissionInfo.scala Outdated

Ngone51 reviewed Aug 18, 2020
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/scheduler/ExecutorDecommissionInfo.scala Outdated

agrawaldevesh force-pushed the consolidate_decom_state branch from 97ad7fe to 7449fa2Compare August 18, 2020 15:59

agrawaldevesh commented Aug 18, 2020
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala Outdated

Ngone51 reviewed Aug 19, 2020
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/scheduler/ExecutorDecommissionInfo.scala Outdated

Ngone51 reviewed Aug 19, 2020
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala Outdated

agrawaldevesh force-pushed the consolidate_decom_state branch from 7449fa2 to 6a5be83Compare August 19, 2020 06:28

cloud-fan reviewed Aug 19, 2020
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala Outdated

cloud-fan reviewed Aug 19, 2020
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala Outdated

agrawaldevesh force-pushed the consolidate_decom_state branch from 6a5be83 to 9222f05Compare August 19, 2020 06:54

agrawaldevesh force-pushed the consolidate_decom_state branch from 9222f05 to ebd8408Compare August 19, 2020 16:30

agrawaldevesh changed the title ~~[SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm~~[SPARK-32643][CORE][K8S] Consolidate state decommissioning in the TaskSchedulerImpl realmAug 19, 2020

agrawaldevesh changed the title ~~[SPARK-32643][CORE][K8S] Consolidate state decommissioning in the TaskSchedulerImpl realm~~[SPARK-32643][CORE][K8s] Consolidate state decommissioning in the TaskSchedulerImpl realmAug 19, 2020

agrawaldevesh force-pushed the consolidate_decom_state branch from ebd8408 to 3541ba9Compare August 19, 2020 22:16

dagrawal3409 added 9 commits August 26, 2020 00:07

Consolidate state decommissioning in the TaskSchedulerImpl realm
6aef647

@Ngone51's comments
e0440b0

@cloud-fan's comments
44dee1b

@Ngone51's comments
a9281d3

Fix build and one more info -> state renaming
807c61b

@holdenk's new comments
b8fef81

Remove message from ExecutorDecommissionState
4b60cef

@Ngone51's test comments
96ba7d5

revert back the extra test to TaskSetManagerSuite as per@Ngone51
29fe131

agrawaldevesh force-pushed the consolidate_decom_state branch from 6b9ded4 to 29fe131Compare August 26, 2020 07:07

asfgit closed this in b786f31Aug 26, 2020

Uh oh!

Conversation

agrawaldevesh commented Aug 18, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

agrawaldevesh commented Aug 18, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Aug 18, 2020

Uh oh!

Ngone51 commented Aug 18, 2020

Uh oh!

SparkQA commented Aug 18, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 18, 2020

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 19, 2020

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 19, 2020

Uh oh!

cloud-fan commented Aug 19, 2020

Uh oh!

SparkQA commented Aug 19, 2020

Uh oh!

SparkQA commented Aug 19, 2020

Uh oh!

holdenk commented Aug 19, 2020

Uh oh!

SparkQA commented Aug 19, 2020

Uh oh!

SparkQA commented Aug 19, 2020

Uh oh!

SparkQA commented Aug 25, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

agrawaldevesh commented Aug 26, 2020

Uh oh!

agrawaldevesh commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

agrawaldevesh commented Aug 26, 2020

Uh oh!

holdenk commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

holdenk commented Aug 26, 2020

Uh oh!

SparkQA commented Aug 26, 2020

Uh oh!

agrawaldevesh commented Aug 27, 2020

Uh oh!

holdenk commented Aug 27, 2020

Uh oh!

Reviewers

Assignees

agrawaldevesh commented Aug 18, 2020•
edited
Loading