SPARK-2791: Fix committing, reverting and state tracking in shuffle file consolidation by aarondav · Pull Request #1678 · apache/spark

aarondav · 2014-07-31T02:01:01Z

All changes from this PR are by @mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle file consolidation that @mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1.

This patch is not intended as a replacement for #1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests.

If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable option.

SparkQA · 2014-07-31T02:03:58Z

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17535/consoleFull

SparkQA · 2014-07-31T02:07:20Z

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17535/consoleFull

@mridulm

All changes from this PR are by @mridulm and are drawn from his work in apache#1609. This patch is intended to fix all major issues related to shuffle file consolidation that @mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1. This patch is **not** intended as a replacement for apache#1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests. If it is feasible to merge apache#1609 for the 1.1 deadline, that is a preferable option.

SparkQA · 2014-07-31T02:18:52Z

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17537/consoleFull

aarondav · 2014-07-31T02:19:40Z

@mridulm Please take a look if possible.

@mateiz This now interacts with the ExternalSorter stuff, and it's possible it partially helps fix a serialization bug (since some serializers apparently write things during a close() after a flush()).

witgo · 2014-07-31T02:29:39Z

We can also merge it to 1.0 branch.

mateiz · 2014-07-31T02:32:53Z

That's true, this is small enough to add back into branch-1.0 too. We should see if we can do the same for the rest of #1609 too (ideally I would like to merge that whole PR into branch-1.0, but I'm not sure how easy that is).

SparkQA · 2014-07-31T04:12:14Z

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17537/consoleFull

SparkQA · 2014-07-31T05:44:14Z

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17553/consoleFull

SparkQA · 2014-07-31T06:28:27Z

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17553/consoleFull

aarondav · 2014-07-31T06:31:14Z

Jenkins, retest this please.

SparkQA · 2014-07-31T06:34:09Z

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17565/consoleFull

SparkQA · 2014-07-31T07:22:19Z

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17565/consoleFull

aarondav · 2014-07-31T16:49:00Z

Both prior failures are due to being unable to bind to ports, which sounds like leftover state from other jenkins runs (especially since they were two entirely disjoint sets of failed tests related to port binding).

aarondav · 2014-07-31T16:49:08Z

Jenkins, retest this please.

SparkQA · 2014-07-31T16:53:58Z

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17585/consoleFull

SparkQA · 2014-07-31T17:49:14Z

QA results for PR 1678:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17585/consoleFull

mridulm · 2014-08-01T04:58:59Z

revert can throw exception : which will cause other writers to not revert.
We need to wrap it in try/catch, log and continue

Revert actually doesn't throw, per its (updated) comment.

mateiz · 2014-08-01T18:49:49Z

Jenkins, test this please

SparkQA · 2014-08-01T18:54:17Z

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17689/consoleFull

SparkQA · 2014-08-01T19:52:55Z

QA results for PR 1678:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17689/consoleFull

mateiz · 2014-08-01T20:24:05Z

@aarondav can you update the name of this PR to reference this sub-task JIRA instead: https://issues.apache.org/jira/browse/SPARK-2791?

aarondav · 2014-08-01T20:45:01Z

Updated title.

mateiz · 2014-08-01T20:54:47Z

Alright, going to merge it then. The changes look good to me.

…ile consolidation All changes from this PR are by mridulm and are drawn from his work in apache#1609. This patch is intended to fix all major issues related to shuffle file consolidation that mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1. This patch is **not** intended as a replacement for apache#1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests. If it is feasible to merge apache#1609 for the 1.1 deadline, that is a preferable option. Author: Aaron Davidson <aaron@databricks.com> Closesapache#1678 from aarondav/consol and squashes the following commits: 53b3f6d [Aaron Davidson] Correct behavior when writing unopened file 701d045 [Aaron Davidson] Rebase with sort-based shuffle 9160149 [Aaron Davidson] SPARK-2532: Minimal shuffle consolidation fixes

JoshRosen · 2015-05-07T18:03:37Z

This catches Exception but it could catch IOException instead.

Co-authored-by: Russell Spitzer <russell.spitzer@gmail.com>

aarondav mentioned this pull request Jul 31, 2014
[SPARK-2532] Consolidated shuffle fixes #1609
Closed
5 tasks

aarondav added 2 commits July 30, 2014 19:08

Rebase with sort-based shuffle
701d045

Correct behavior when writing unopened file
53b3f6d

mridulm reviewed Aug 1, 2014
View reviewed changes

aarondav changed the title ~~SPARK-2532: Minimal shuffle consolidation fixes~~SPARK-2791: Fix committing, reverting and state tracking in shuffle file consolidationAug 1, 2014

asfgit closed this in 78f2af5Aug 1, 2014

JoshRosen reviewed May 7, 2015
View reviewed changes

davies mentioned this pull request Nov 13, 2015
[SPARK-8029] robust shuffle writer (for 1.5 branch) #9686
Closed

sunchao pushed a commit to sunchao/spark that referenced this pull request Jun 2, 2023
rdar://105287150 Update ADT to 1.1.15 (apache#1678)
8ff4a73
Co-authored-by: Russell Spitzer <russell.spitzer@gmail.com>

Uh oh!

Conversation

aarondav commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

aarondav commented Jul 31, 2014

Uh oh!

witgo commented Jul 31, 2014

Uh oh!

mateiz commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

aarondav commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

aarondav commented Jul 31, 2014

Uh oh!

aarondav commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

SparkQA commented Jul 31, 2014

Uh oh!

mridulmAug 1, 2014

Choose a reason for hiding this comment

Uh oh!

aarondavAug 1, 2014

Choose a reason for hiding this comment

Uh oh!

mateiz commented Aug 1, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

SparkQA commented Aug 1, 2014

Uh oh!

mateiz commented Aug 1, 2014

Uh oh!

aarondav commented Aug 1, 2014

Uh oh!

mateiz commented Aug 1, 2014

Uh oh!

JoshRosenMay 7, 2015

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants