[SPARK-19306][Core] Fix inconsistent state in DiskBlockObject when expection occurred #16657
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: I837b5135dd67034d74a9832133dc29800c88f089
|
CC @mridulm please help to review, thanks a lot. |
|
Test build #71712 has finished for PR 16657 at commit
|
|
Jenkins, retest this please. |
mridulm
reviewed
Jan 20, 2017
| } catch { | ||
| case e: Exception => | ||
| logError("Uncaught exception while reverting partial writes to file " + file, e) | ||
| file | ||
| } finally { | ||
| truncateStream.close() |
There was a problem hiding this comment.
Another option is to use Utils.tryWithSafeFinally and do the if block with closeResources in the block with truncate in the finally block
There was a problem hiding this comment.
You actually can't call close() here without a null check.
I'd also move the file return out of the try/catch block altogether.
There was a problem hiding this comment.
Sorry about it. I will fix it.
|
Test build #71716 has finished for PR 16657 at commit
|
Change-Id: I6f88a75cf6467508c77cbfc28c25e4fc7c172373
|
Test build #71800 has started for PR 16657 at commit |
|
LGTM, will wait for @vanzin's comments too. |
mridulm
approved these changes
Jan 22, 2017
|
Jenkins, retest this please. |
|
Test build #71801 has finished for PR 16657 at commit
|
|
LGTM. Merging to master / 2.1. |
asfgit
pushed a commit
that referenced
this pull request
Jan 23, 2017
…pection occurred ## What changes were proposed in this pull request? In `DiskBlockObjectWriter`, when some errors happened during writing, it will call `revertPartialWritesAndClose`, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue. ## How was this patch tested? Existing test. Author: jerryshao <sshao@hortonworks.com> Closes #16657 from jerryshao/SPARK-19306. (cherry picked from commit e497472) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…pection occurred ## What changes were proposed in this pull request? In `DiskBlockObjectWriter`, when some errors happened during writing, it will call `revertPartialWritesAndClose`, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue. ## How was this patch tested? Existing test. Author: jerryshao <sshao@hortonworks.com> Closes apache#16657 from jerryshao/SPARK-19306.
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
…pection occurred ## What changes were proposed in this pull request? In `DiskBlockObjectWriter`, when some errors happened during writing, it will call `revertPartialWritesAndClose`, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue. ## How was this patch tested? Existing test. Author: jerryshao <sshao@hortonworks.com> Closes apache#16657 from jerryshao/SPARK-19306.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In
DiskBlockObjectWriter, when some errors happened during writing, it will callrevertPartialWritesAndClose, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue.How was this patch tested?
Existing test.