HADOOP-16769. LocalDirAllocator to provide diagnostics when file creation fails #1768
+25
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n file creation is not successuful
bgaborg
reviewed
Dec 18, 2019
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalDirAllocator.java
Show resolved
Hide resolved
…n file creation is not successuful
…n file creation is not successuful
|
💔 -1 overall
This message was automatically generated. |
steveloughran
changed the title
steveloughran
requested changes
Jan 2, 2020
There was a problem hiding this comment.
Now, there's another failure mode: no write access to any of the specified dirs.
If that's the situation, we should be able to identify and report it as well. Indeed, now that we have being helpful on disk capacity -we probably need to differentiate the other failure mode to avoid confusion. Otherwise people who encounter permissions problems we misled into thinking its disk capacity.
Not sure the best approach here. We would probably need to keep the DiskErrorException from createPath() and use that as the inner cause of the new failure. That is: move the catch() clause up to the caller where the exception can be cached for later use.
This is probably broadly useful as the other failure modes which are probably worth reporting.
Goal: there's enough information in the strings and stack traces to identify the root cause without having to chase into the logs of machines.
| String dir0 = buildBufferDir(ROOT, 0); | ||
| String dir1 = buildBufferDir(ROOT, 1); | ||
| conf.set(CONTEXT, dir0 + "," + dir1); | ||
| try { |
There was a problem hiding this comment.
use LambaTestUtils.intercept, which will do the catch, reporting on failure and error message checks
| @@ -532,4 +532,23 @@ public void testGetLocalPathForWriteForInvalidPaths() throws Exception { | |||
| } | |||
| } | |||
|
|
|||
| /** | |||
| * Test to check the LocalDirAllocation for the less space HADOOP-16769 | |||
There was a problem hiding this comment.
add trailing "." for javadocs
| String dir1 = buildBufferDir(ROOT, 1); | ||
| conf.set(CONTEXT, dir0 + "," + dir1); | ||
| try { | ||
| dirAllocator.getLocalPathForWrite("p1/x", 3_000_000_000_000L, conf); |
There was a problem hiding this comment.
what's going to happen on a disk with >3TB of capacity? Should we go for a bigger number?
There was a problem hiding this comment.
Sorry I missed updating the pull request. I have a new code change that uses Long.MAX_VALUE, instead of this hardcoded number and then use a regex to match the error message. I will create a new pull request
|
I will address the above change to catch and throw the two errors, so they are nested into one another, as part of the new pull request. Thanks! |
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.