Skip to content

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer - #39819

Closed
wayneguow wants to merge 11 commits into
apache:masterfrom
wayneguow:shuffle_output_buffer
Closed

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer#39819
wayneguow wants to merge 11 commits into
apache:masterfrom
wayneguow:shuffle_output_buffer

Conversation

@wayneguow

@wayneguowwayneguow commented Jan 31, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config spark.shuffle.localDisk.file.output.buffer instead.

Why are the changes needed?

The old config is desgined to be used in UnsafeShuffleWriter, but now it has been used in all local shuffle writers through LocalDiskShuffleMapOutputWriter, introduced by #25007.

Does this PR introduce any user-facing change?

Old still works, advised to use new.

How was this patch tested?

Passed existing tests.

@wayneguowwayneguow changed the title [SPARK-42252][Core] Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config[SPARK-42252][CORE] Deprecate spark.shuffle.unsafe.file.output.buffer and add a new configJan 31, 2023
@wayneguow

wayneguow commented Jan 31, 2023

Copy link
Copy Markdown
ContributorAuthor

cc @mccheah and @HyukjinKwon

@wayneguow
wayneguowforce-pushed the shuffle_output_buffer branch from 4696a36 to bd0c1b6CompareFebruary 3, 2023 13:04
dongjoon-hyun
dongjoon-hyun previously requested changes Feb 3, 2023
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you don't mind, let's target this proposal for Apache Spark 3.5.0.

@wayneguow

wayneguow commented Feb 4, 2023

Copy link
Copy Markdown
ContributorAuthor

If you don't mind, let's target this proposal for Apache Spark 3.5.0.

Ok, it's good. I will make some changes.

@wayneguow

wayneguow commented Feb 7, 2023

Copy link
Copy Markdown
ContributorAuthor

Gentle ping @dongjoon-hyun, the target version of this proposal has been updated. Could you go on reviewing it when you have time? Thanks.

@dongjoon-hyun

dongjoon-hyun commented May 5, 2023

Copy link
Copy Markdown
Member

Thank you for your patience. Could you rebase this PR?

dongjoon-hyun
dongjoon-hyun reviewed May 5, 2023
View reviewed changes
Comment threaddocs/core-migration-guide.md Outdated

@dongjoon-hyundongjoon-hyunMay 5, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be Since Spark 3.5.

@wayneguowwayneguowMay 6, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm here, updated it, and waiting for GA.

@wayneguow
wayneguowforce-pushed the shuffle_output_buffer branch from fe8729d to 6fafbc5CompareMay 6, 2023 08:08
@dongjoon-hyundongjoon-hyun changed the title [SPARK-42252][CORE] Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.bufferMay 7, 2023
dongjoon-hyun
dongjoon-hyun reviewed May 12, 2023
View reviewed changes
Comment threaddocs/core-migration-guide.md Outdated

@dongjoon-hyundongjoon-hyunMay 7, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove thought still works. deprecated means it still works.

@wayneguowwayneguowMay 12, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solved

@HyukjinKwon

HyukjinKwon commented May 15, 2023

Copy link
Copy Markdown
Member

@wayneguow which PR made this change:

The old config is desgined to be used in UnsafeShuffleWriter, but now it has been used in all local shuffle writers through LocalDiskShuffleMapOutputWriter.

? Would be easier for me to review if you can point it out.

@wayneguow

wayneguow commented May 15, 2023

Copy link
Copy Markdown
ContributorAuthor

@wayneguow which PR made this change:

The old config is desgined to be used in UnsafeShuffleWriter, but now it has been used in all local shuffle writers through LocalDiskShuffleMapOutputWriter.

? Would be easier for me to review if you can point it out.

@HyukjinKwon The PR is #25007 .

dongjoon-hyun
dongjoon-hyun approved these changes May 18, 2023
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM from my side.

cc @HyukjinKwon , @cloud-fan , @Ngone51 , @mridulm

@github-actions

github-actionsBot commented Aug 27, 2023

Copy link
Copy Markdown

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

@dongjoon-hyun

dongjoon-hyun commented Aug 28, 2023

Copy link
Copy Markdown
Member

Too bad. Sorry but it seems that we forgot this PR, @wayneguow .

@dongjoon-hyun

dongjoon-hyun commented Aug 28, 2023

Copy link
Copy Markdown
Member

It seems too late because Apache Spark 3.5 is RC3 stage. If we want this still, we can do for Apache Spark 4.0.0.

@wayneguow
wayneguowforce-pushed the shuffle_output_buffer branch from 5f19b3d to 02843cfCompareMay 30, 2024 12:01
wayneguow
wayneguow commented May 30, 2024
View reviewed changes
Comment threaddocs/core-migration-guide.md Outdated
@wayneguow

wayneguow commented May 30, 2024

Copy link
Copy Markdown
ContributorAuthor

Gentle ping @dongjoon-hyun , a pr from some time ago.

@LuciferYang

LuciferYang commented Jun 4, 2024

Copy link
Copy Markdown
Contributor

friendly ping @yaooqinn Do you have time to help take a look?

@yaooqinn

yaooqinn commented Jun 14, 2024

Copy link
Copy Markdown
Member

Merged to master, thank you @dongjoon-hyun@LuciferYang@wayneguow

@wayneguow
wayneguow deleted the shuffle_output_buffer branch February 11, 2025 04:25
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LuciferYangLuciferYangLuciferYang approved these changes
@yaooqinnyaooqinnyaooqinn approved these changes
@dongjoon-hyundongjoon-hyundongjoon-hyun approved these changes

Assignees

No one assigned

Labels

COREDOCS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wayneguow@dongjoon-hyun@HyukjinKwon@LuciferYang@yaooqinn