Uh oh!
There was an error while loading. Please reload this page.
[SPARK-19306][Core] Fix inconsistent state in DiskBlockObject when expection occurred - #16657
Closed
jerryshao wants to merge 2 commits into
Closed
[SPARK-19306][Core] Fix inconsistent state in DiskBlockObject when expection occurred#16657jerryshao wants to merge 2 commits into
jerryshao wants to merge 2 commits into
Conversation
Change-Id: I837b5135dd67034d74a9832133dc29800c88f089
jerryshao
commented
Jan 20, 2017
ContributorAuthor
CC @mridulm please help to review, thanks a lot. |
SparkQA
commented
Jan 20, 2017
Test build #71712 has finished for PR 16657 at commit
|
jerryshao
commented
Jan 20, 2017
ContributorAuthor
Jenkins, retest this please. |
mridulm
reviewed
Jan 20, 2017
| logError("Uncaught exception while reverting partial writes to file " + file, e) | ||
| file | ||
| } finally { | ||
| truncateStream.close() |
Contributor
There was a problem hiding this comment.
Another option is to use Utils.tryWithSafeFinally and do the if block with closeResources in the block with truncate in the finally block
Contributor
There was a problem hiding this comment.
You actually can't call close() here without a null check.
I'd also move the file return out of the try/catch block altogether.
ContributorAuthor
There was a problem hiding this comment.
Sorry about it. I will fix it.
SparkQA
commented
Jan 20, 2017
Test build #71716 has finished for PR 16657 at commit
|
Change-Id: I6f88a75cf6467508c77cbfc28c25e4fc7c172373
SparkQA
commented
Jan 22, 2017
Test build #71800 has started for PR 16657 at commit |
mridulm
commented
Jan 22, 2017
Contributor
LGTM, will wait for @vanzin's comments too. |
mridulm
approved these changes
Jan 22, 2017
jerryshao
commented
Jan 22, 2017
ContributorAuthor
Jenkins, retest this please. |
SparkQA
commented
Jan 22, 2017
Test build #71801 has finished for PR 16657 at commit
|
vanzin
commented
Jan 23, 2017
Contributor
LGTM. Merging to master / 2.1. |
asfgit pushed a commit
that referenced
this pull request
Jan 23, 2017
…pection occurred ## What changes were proposed in this pull request? In `DiskBlockObjectWriter`, when some errors happened during writing, it will call `revertPartialWritesAndClose`, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue. ## How was this patch tested? Existing test. Author: jerryshao <sshao@hortonworks.com> Closes#16657 from jerryshao/SPARK-19306. (cherry picked from commit e497472) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
uzadude pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…pection occurred ## What changes were proposed in this pull request? In `DiskBlockObjectWriter`, when some errors happened during writing, it will call `revertPartialWritesAndClose`, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue. ## How was this patch tested? Existing test. Author: jerryshao <sshao@hortonworks.com> Closesapache#16657 from jerryshao/SPARK-19306.
cmonkey pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
…pection occurred ## What changes were proposed in this pull request? In `DiskBlockObjectWriter`, when some errors happened during writing, it will call `revertPartialWritesAndClose`, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue. ## How was this patch tested? Existing test. Author: jerryshao <sshao@hortonworks.com> Closesapache#16657 from jerryshao/SPARK-19306.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In
DiskBlockObjectWriter, when some errors happened during writing, it will callrevertPartialWritesAndClose, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue.How was this patch tested?
Existing test.