HDFS-17044. Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode. #5735
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…block size to NO_ACK
|
💔 -1 overall
This message was automatically generated. |
|
@haiyang1987 , Hi, haiyang bro, please correct me if i understand wrongly. Hoping to receive your response, thanks. |
|
@hfutatzhanghb Hi, bro thanks for your comment. I will add UT try to illustrate this case. |
@haiyang1987 Thanks bro for replying. Got it~ |
|
Great catch here. One nit comment, just suggest to add some explicit annotations which avoid to useless notification from datanode after delete. And, add new UT will be better here. |
HI @Hexiaoqiao Sir, thanks for your comment. |
Hexiaoqiao
reviewed
Jun 17, 2023
| @@ -3295,8 +3295,11 @@ private BlockInfo processReportedBlock( | |||
| BlockInfo storedBlock = getStoredBlock(block); | |||
| if(storedBlock == null) { | |||
| // If blocksMap does not contain reported block id, | |||
| // the replica should be removed from the data-node. | |||
| toInvalidate.add(new Block(block)); | |||
| // the replica should be removed from the data-node, | |||
There was a problem hiding this comment.
The replica should be removed from Datanode, and set NumBytes to BlockCommand.No_ACK to avoid useless report to NameNode from Datanode when complete to process it.
There was a problem hiding this comment.
Sorry for the late reply.
Noted, I will update PR today, Thanks.
There was a problem hiding this comment.
Sir, please help reivew it again, Thanks.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
The failed unit test seems unrelated to the change. |
|
@Hexiaoqiao Hi Sir, please help review it again when you have time. Thanks a lot~. |
Hexiaoqiao
approved these changes
Jun 26, 2023
Hexiaoqiao
changed the title
|
Committed to trunk. Thanks @haiyang1987 for your contributions, And thanks @hfutatzhanghb too. |
|
Thanks @Hexiaoqiao @hfutatzhanghb help review and merge it. |
jiajunmao
pushed a commit
to jiajunmao/hadoop-MLEC
that referenced
this pull request
Feb 6, 2024
… IBR to avoid useless report from DataNode. (apache#5735). Contributed by Haiyang Hu. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
https://issues.apache.org/jira/browse/HDFS-17044
When NameNode processes DataNode increment or full block report, if block is not in the blocks map, it will be added to invalidate and the replica should be removed from the data-node, and the block size should be to set NO_ACK, should be reduce some useless DataNode block reports.