HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() #2785
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
virajjasani
force-pushed
the
HDFS-15904-trunk
branch
from
efb276c to
1cb6258
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
liuml07
approved these changes
Mar 19, 2021
| @@ -2297,7 +2297,9 @@ public void testBalancerWithSortTopNodes() throws Exception { | |||
| maxUsage = Math.max(maxUsage, datanodeReport[i].getDfsUsed()); | |||
| } | |||
|
|
|||
| assertEquals(200, balancerResult.bytesAlreadyMoved); | |||
| // Either 2 blocks of 100+100 bytes or 3 blocks of 100+100+50 bytes | |||
There was a problem hiding this comment.
Could add some explanation why this would happen.
The 95% usage DN will have 9 blocks of 100 bytes and 1 block of 50 byte - all for the same file. The HDFS balancer will choose a block to move from this node randomly. More likely it will be 100B block. Since that is greater than DFS_BALANCER_MAX_SIZE_TO_MOVE_KEY which is 99L (see above settings), it will stop here. Total bytes moved from this 95% DN will be 1 block and hence 100B.
However, chances are the first block to move from this 95% DN is the 50B block. After this block being moved, the total moved size 50B is smaller than DFS_BALANCER_MAX_SIZE_TO_MOVE_KEY , it will try to move another block. The second block will always be 100 bytes. So total bytes moved from this 95% DN will be 2 blocks and hence 150B (100B + 50B).
Please reword or rephrase this as comment before this assertion so readers can have more context without thinking too much again.
Thanks,
There was a problem hiding this comment.
Sure thing, let me provide detailed comment as per your suggestion.
Thanks
virajjasani
changed the title
virajjasani
force-pushed
the
HDFS-15904-trunk
branch
from
1cb6258 to
5dc3756
Compare
|
Will commit after a good QA. Thanks |
|
💔 -1 overall
This message was automatically generated. |
|
Majority test failures are due to timeout. Let me trigger build one more time? |
ayushtkn
approved these changes
Mar 19, 2021
There was a problem hiding this comment.
Thanx @virajjasani for the fix, changes LGTM.
Have triggered the build again.
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2785/5/
|
💔 -1 overall
This message was automatically generated. |
|
comments: failing tests were not related or due to testing machine out of memory error. We have so many intermittent failures in HDFS tests. Glad to see bug fixes like this. Thank you! |
kiran-maturi
pushed a commit
to kiran-maturi/hadoop
that referenced
this pull request
Nov 24, 2021
…ache#2785) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.