Skip to content

[SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo - #38030

Closed
bozhang2820 wants to merge 4 commits into
apache:masterfrom
bozhang2820:spark-40596
Closed

[SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo#38030
bozhang2820 wants to merge 4 commits into
apache:masterfrom
bozhang2820:spark-40596

Conversation

@bozhang2820

@bozhang2820bozhang2820 commented Sep 28, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This change populates ExecutorDecommission with messages in ExecutorDecommissionInfo.

Why are the changes needed?

Currently the message in ExecutorDecommission is a fixed value ("Executor decommission."), so it is the same for all cases, e.g. spot instance interruptions and auto-scaling down. With this change we can better differentiate those cases.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added a unit test.

@bozhang2820bozhang2820 changed the title [SPARK-40596] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo[SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfoSep 28, 2022
@mridulm

mridulm commented Sep 28, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

How/where are we using the message ?

@mridulm

mridulm commented Sep 28, 2022

Copy link
Copy Markdown
Contributor

Ok, so this is mainly to propagate in SparkListenerExecutorRemoved, sounds reasonable.

+CC @dongjoon-hyun

@dongjoon-hyun

dongjoon-hyun commented Sep 28, 2022

Copy link
Copy Markdown
Member

Thank you, @mridulm .

dongjoon-hyun
dongjoon-hyun reviewed Sep 28, 2022
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/scheduler/MapStatus.scala Outdated
dongjoon-hyun
dongjoon-hyun previously requested changes Sep 28, 2022
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert the change on MapStatus.

Ngone51
Ngone51 reviewed Sep 29, 2022
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala Outdated
Ngone51
Ngone51 reviewed Sep 29, 2022
View reviewed changes
Comment threadcore/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionIntegrationSuite.scala Outdated
@dongjoon-hyun

dongjoon-hyun commented Oct 4, 2022

Copy link
Copy Markdown
Member

Could you address the review comments, @bozhang2820 ?

@bozhang2820

bozhang2820 commented Oct 8, 2022

Copy link
Copy Markdown
ContributorAuthor

Could you address the review comments, @bozhang2820 ?

Sorry for the late response. I was on vacation last week. Addressed the review comments.

@bozhang2820

bozhang2820 commented Oct 9, 2022

Copy link
Copy Markdown
ContributorAuthor

Build passed with a retry: https://github.com/bozhang2820/spark/actions/runs/3209525689/jobs/5251496034

dongjoon-hyun
dongjoon-hyun approved these changes Oct 9, 2022
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for update, @bozhang2820 . It looks good to me.

What do you think about the Today's patch, @mridulm , @Ngone51 , @xuanyuanking?

Ngone51
Ngone51 approved these changes Oct 10, 2022
View reviewed changes

@Ngone51Ngone51 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

xuanyuanking
xuanyuanking approved these changes Oct 10, 2022
View reviewed changes

@xuanyuankingxuanyuanking left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Ngone51

Ngone51 commented Oct 11, 2022

Copy link
Copy Markdown
Member

Thanks, merged to master!

@dongjoon-hyun

dongjoon-hyun commented Oct 11, 2022

Copy link
Copy Markdown
Member

Thank you all! :)

@bozhang2820
bozhang2820 deleted the spark-40596 branch October 11, 2022 11:02
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xuanyuankingxuanyuankingxuanyuanking approved these changes
@dongjoon-hyundongjoon-hyundongjoon-hyun approved these changes
@Ngone51Ngone51Ngone51 approved these changes

Assignees

No one assigned

Labels

CORE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bozhang2820@mridulm@dongjoon-hyun@Ngone51@xuanyuanking