Skip to content

HDDS-1146. Adding container related metrics in SCM. #1541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 27, 2019

Conversation

bharatviswa504
Copy link
Contributor

No description provided.

Copy link
Contributor

@anuengineer anuengineer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, there is a minor comment. Feel free to commit once you get a CI run.

* @param isFullReport
* @param success
*/
void notifyContainerReportProcessing(boolean isFullReport, boolean success);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure, But I think we have a Enum that indicates if the report is full or partial. If there is already one, then we might want to use it. Otherwise, I am okay with the boolean.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From proto, there is no enum. But i see we have TypedEvent
TypedEvent
INCREMENTAL_CONTAINER_REPORT = new TypedEvent<>(
IncrementalContainerReportFromDatanode.class,
"Incremental_Container_Report")
TypedEvent CONTAINER_REPORT =
new TypedEvent<>(ContainerReportFromDatanode.class, "Container_Report");

Do you mean we want to use them instead of boolean?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 1407 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 40 Maven dependency ordering for branch
-1 mvninstall 48 hadoop-hdds in trunk failed.
-1 mvninstall 47 hadoop-ozone in trunk failed.
-1 compile 22 hadoop-hdds in trunk failed.
-1 compile 14 hadoop-ozone in trunk failed.
+1 checkstyle 68 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 949 branch has no errors when building and testing our client artifacts.
-1 javadoc 21 hadoop-hdds in trunk failed.
-1 javadoc 19 hadoop-ozone in trunk failed.
0 spotbugs 1049 Used deprecated FindBugs config; considering switching to SpotBugs.
-1 findbugs 36 hadoop-hdds in trunk failed.
-1 findbugs 19 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
0 mvndep 28 Maven dependency ordering for patch
-1 mvninstall 36 hadoop-hdds in the patch failed.
-1 mvninstall 38 hadoop-ozone in the patch failed.
-1 compile 24 hadoop-hdds in the patch failed.
-1 compile 16 hadoop-ozone in the patch failed.
-1 javac 24 hadoop-hdds in the patch failed.
-1 javac 16 hadoop-ozone in the patch failed.
+1 checkstyle 61 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 773 patch has no errors when building and testing our client artifacts.
-1 javadoc 20 hadoop-hdds in the patch failed.
-1 javadoc 17 hadoop-ozone in the patch failed.
-1 findbugs 32 hadoop-hdds in the patch failed.
-1 findbugs 18 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 27 hadoop-hdds in the patch failed.
-1 unit 26 hadoop-ozone in the patch failed.
+1 asflicense 32 The patch does not generate ASF License warnings.
3947
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/Dockerfile
GITHUB PR #1541
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 13e1622b27d9 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 13b427f
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/testReport/
Max. process+thread count 305 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/2/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 1737 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 77 Maven dependency ordering for branch
-1 mvninstall 48 hadoop-hdds in trunk failed.
-1 mvninstall 48 hadoop-ozone in trunk failed.
-1 compile 22 hadoop-hdds in trunk failed.
-1 compile 14 hadoop-ozone in trunk failed.
+1 checkstyle 68 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 947 branch has no errors when building and testing our client artifacts.
-1 javadoc 23 hadoop-hdds in trunk failed.
-1 javadoc 18 hadoop-ozone in trunk failed.
0 spotbugs 1050 Used deprecated FindBugs config; considering switching to SpotBugs.
-1 findbugs 38 hadoop-hdds in trunk failed.
-1 findbugs 19 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
0 mvndep 29 Maven dependency ordering for patch
-1 mvninstall 35 hadoop-hdds in the patch failed.
-1 mvninstall 39 hadoop-ozone in the patch failed.
-1 compile 23 hadoop-hdds in the patch failed.
-1 compile 18 hadoop-ozone in the patch failed.
-1 javac 23 hadoop-hdds in the patch failed.
-1 javac 18 hadoop-ozone in the patch failed.
+1 checkstyle 62 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 798 patch has no errors when building and testing our client artifacts.
-1 javadoc 19 hadoop-hdds in the patch failed.
-1 javadoc 18 hadoop-ozone in the patch failed.
-1 findbugs 31 hadoop-hdds in the patch failed.
-1 findbugs 19 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 28 hadoop-hdds in the patch failed.
-1 unit 24 hadoop-ozone in the patch failed.
+1 asflicense 28 The patch does not generate ASF License warnings.
4336
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/Dockerfile
GITHUB PR #1541
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 8278c62831ff 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 13b427f
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/testReport/
Max. process+thread count 305 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1541/1/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@avijayanhwx
Copy link
Contributor

avijayanhwx commented Sep 27, 2019

@bharatviswa504 If you don't mind, can you tell me where a metric like numContainerReportsSuccess would be useful? Failures I guess may be useful.

LOG.error("Exception while processing ICR for container {}",
replicaProto.getContainerID());
}
}

if (success) {
getContainerManager().notifyContainerReportProcessing(false, true);
} else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason we are not doing getContainerManager().notifyContainerReportProcessing(false, success) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do that, just due to copy/paste from ICR code it is like this. (End result is same)
Let me know if you want to change it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, we should change it to 1 line instead of if-else.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, will change this in later jira. For now we shall go with the current way.

@bharatviswa504
Copy link
Contributor Author

bharatviswa504 commented Sep 27, 2019

@bharatviswa504 If you don't mind, can you tell me where a metric like numContainerReportsSuccess would be useful? Failures I guess may be useful.

If the user wants to know numContainerReportsProcessed successfully in system. And he wants to compare with failure count, then it might be useful.

And also I got to know yesterday it would be good to have metrics in the system for some actions happened/performed.

@bharatviswa504
Copy link
Contributor Author

Test failures are not related to this patch.
Thank You @anuengineer and @avijayanhwx for the review.
I will commit this to the trunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants