Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-2267. Container metadata scanner interval mismatch #1624

Closed
wants to merge 1 commit into from

Conversation

adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

Fix time unit mismatch in container metadata scanner. Elapsed time is measured in nanoseconds, needs to be converted to milliseconds.

https://issues.apache.org/jira/browse/HDDS-2267

How was this patch tested?

Tested on ozone docker-compose cluster.

datanode_1  | 2019-10-09 06:18:00 INFO  ContainerMetadataScanner:60 - Background ContainerMetadataScanner starting up
datanode_1  | 2019-10-09 06:18:00 INFO  ContainerMetadataScanner:88 - Completed an iteration of container metadata scrubber in 0 minutes. Number of  iterations (since the data-node restart) : 1, Number of containers scanned in this iteration : 0, Number of unhealthy containers found in this iteration : 0
datanode_1  | 2019-10-09 06:19:00 INFO  ContainerMetadataScanner:88 - Completed an iteration of container metadata scrubber in 0 minutes. Number of  iterations (since the data-node restart) : 2, Number of containers scanned in this iteration : 2, Number of unhealthy containers found in this iteration : 0
...
datanode_1  | STARTUP_MSG: Starting HddsDatanodeService
datanode_1  | 2019-10-09 06:23:52 INFO  ContainerMetadataScanner:60 - Background ContainerMetadataScanner starting up
datanode_1  | 2019-10-09 06:23:52 INFO  ContainerMetadataScanner:88 - Completed an iteration of container metadata scrubber in 0 minutes. Number of  iterations (since the data-node restart) : 1, Number of containers scanned in this iteration : 2, Number of unhealthy containers found in this iteration : 0
datanode_1  | 2019-10-09 06:24:52 INFO  ContainerMetadataScanner:88 - Completed an iteration of container metadata scrubber in 0 minutes. Number of  iterations (since the data-node restart) : 2, Number of containers scanned in this iteration : 2, Number of unhealthy containers found in this iteration : 0

@adoroszlai
Copy link
Contributor Author

/label ozone

@elek elek added the ozone label Oct 9, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 85 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 mvninstall 52 hadoop-hdds in trunk failed.
-1 mvninstall 39 hadoop-ozone in trunk failed.
-1 compile 18 hadoop-hdds in trunk failed.
-1 compile 13 hadoop-ozone in trunk failed.
+1 checkstyle 63 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 957 branch has no errors when building and testing our client artifacts.
-1 javadoc 19 hadoop-hdds in trunk failed.
-1 javadoc 16 hadoop-ozone in trunk failed.
0 spotbugs 1044 Used deprecated FindBugs config; considering switching to SpotBugs.
-1 findbugs 31 hadoop-hdds in trunk failed.
-1 findbugs 16 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
-1 mvninstall 32 hadoop-hdds in the patch failed.
-1 mvninstall 35 hadoop-ozone in the patch failed.
-1 compile 22 hadoop-hdds in the patch failed.
-1 compile 16 hadoop-ozone in the patch failed.
-1 javac 22 hadoop-hdds in the patch failed.
-1 javac 16 hadoop-ozone in the patch failed.
+1 checkstyle 51 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 798 patch has no errors when building and testing our client artifacts.
-1 javadoc 18 hadoop-hdds in the patch failed.
-1 javadoc 16 hadoop-ozone in the patch failed.
-1 findbugs 28 hadoop-hdds in the patch failed.
-1 findbugs 18 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 24 hadoop-hdds in the patch failed.
-1 unit 21 hadoop-ozone in the patch failed.
+1 asflicense 30 The patch does not generate ASF License warnings.
2526
Subsystem Report/Notes
Docker Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/Dockerfile
GITHUB PR #1624
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 3c965ea1d7b1 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1d27930
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/testReport/
Max. process+thread count 306 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1624/1/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to fix it @adoroszlai. Nice catch.

As the millisecond resolution is enough I think it can be more safer to use type-safe java API:

Sg like:

Instant start = Instant.now();

...

nextCheck = start.plus(metadataScanInterval, ChronoUnit.SECONDS)
remainingMs = Instant.now().until(nextCheck, ChronoUnit.MILLIS)
Time.sleep(remainingMs)

But I am also fine with the current patch as it's definitely better than the earlier code ;-)

@elek
Copy link
Member

elek commented Oct 13, 2019

Thank you very much to open this pull request.

During the weekend the Ozone source code has been moved out from apache/hadoop repository to apache/hadoop-ozone repository.

This git commits are rewritten, but the branch of this pull request is also transformed (state of Saturday morning), you can use the new, migrated branch to recreate this pull request.

Your pull request is important for us: Can you please re-create your pull request in the new repository?

1. Create a new fork of https://github.com/apache/hadoop-ozone

2. Clone it and have both your fork and the apache repo as remotes:

git clone git@github.com:adoroszlai/hadoop-ozone.git
cd hadoop-ozone
git remote add apache git@github.com:apache/hadoop-ozone.git
git fetch apache

3. Fetch your migrated branch and push it to your fork.

git checkout -b HDDS-2267 apache/HDDS-2267
git push origin HDDS-2267

4. And create the new pull request on the new repository.

https://github.com/apache/hadoop-ozone/compare/master...adoroszlai:HDDS-2267?expand=1

If you need more information, please check this wiki page or contact with me (my github user name + apache.org).

Thank you, and sorry for the inconvenience.

@adoroszlai
Copy link
Contributor Author

Moved to apache/ozone#8

@adoroszlai adoroszlai closed this Oct 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants