Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-15634 [WIP] Invalidate replica on decommissioning/decommissioned… #2388

Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from

Conversation

fengnanli
Copy link
Contributor

… Datanodes

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

return storedBlock;
}

// If there are enough live replicas, start invalidating
// decommissioned + decommissioning replicas
private boolean shouldInvalidateDecommissionedRedundancy(NumberReplicas num,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense. Maybe we should describe some of the JIRA description in this method to explain what we are doing in the high level.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea.

if(storedBlock.getBlockUCState() == BlockUCState.COMMITTED &&

// if block is still under construction, then done for now
if (!storedBlock.isCompleteOrCommitted()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we move this block here?
BTW we can leave it as a single if with a return.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I felt quite confused with the original structure since the early return was put after the statements it is trying to avoid..
I can make it a single return no big deal.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 28m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 29m 41s trunk passed
+1 💚 compile 1m 16s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 compile 1m 9s trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 checkstyle 0m 51s trunk passed
+1 💚 mvnsite 1m 18s trunk passed
+1 💚 shadedclient 16m 8s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 54s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javadoc 1m 26s trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗 spotbugs 3m 2s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 0s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 1m 13s the patch passed
+1 💚 compile 1m 7s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javac 1m 7s the patch passed
+1 💚 compile 1m 5s the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 javac 1m 5s the patch passed
+1 💚 checkstyle 0m 42s the patch passed
+1 💚 mvnsite 1m 11s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 11s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 47s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javadoc 1m 21s the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚 findbugs 2m 58s the patch passed
_ Other Tests _
-1 ❌ unit 95m 11s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 45s The patch does not generate ASF License warnings.
207m 14s
Reason Tests
Failed junit tests hadoop.hdfs.TestFileChecksum
hadoop.hdfs.TestDecommissionWithStriped
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
hadoop.hdfs.server.namenode.TestFsck
hadoop.hdfs.TestFileChecksumCompositeCrc
hadoop.hdfs.TestStripedFileAppend
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2388/1/artifact/out/Dockerfile
GITHUB PR #2388
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 2bbb452b3bd4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8b8c672
Default Java Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2388/1/testReport/
Max. process+thread count 4131 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2388/1/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@fengnanli
Copy link
Contributor Author

Will put another patch with UT soon

@jojochuang jojochuang added the work in progress PRs still Work in Progress; reviews not expected but still welcome label Nov 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
work in progress PRs still Work in Progress; reviews not expected but still welcome
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants