Skip to content

HADOOP-16290. Enable RpcMetrics units to be configurable #3198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jul 20, 2021
Merged

HADOOP-16290. Enable RpcMetrics units to be configurable #3198

merged 5 commits into from
Jul 20, 2021

Conversation

virajjasani
Copy link
Contributor

No description provided.

@virajjasani
Copy link
Contributor Author

Jenkins runs are failing recently, looks like an infra issue based on logs: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3198/1/console

@hadoop-yetus

This comment has been minimized.

Sign in to view
@hadoop-yetus

This comment has been minimized.

Sign in to view
jojochuang
jojochuang reviewed Jul 15, 2021
View reviewed changes
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java Outdated
RpcMetrics m = new RpcMetrics(server, conf);
return DefaultMetricsSystem.instance().register(m.name, null, m);
}

private static void setMetricTimeUnit(Configuration conf) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be error-prone because a process can have multiple Server binding at different ports at the same time.
So metricsTimeUnit is initialized multiple times for each Server creation. Hopefully they end up using the same metric unit. Otherwise this could hypothetically create confusion later on.

Copy link
Contributor Author

@virajjasani virajjasani Jul 15, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, let me make it this way. If server is instance of RPC.Server (which is LimitedPrivate for Commons, HDFS, Yarn, MapReduce), we can make this change so that for anyone extending this Server from downstream application does not interfere with core-site config used in core Hadoop.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor Author

@virajjasani virajjasani Jul 20, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is recently addressed better by keeping metricsTimeUnit as non-static final.

@hadoop-yetus

This comment has been minimized.

Sign in to view
@hadoop-yetus

This comment has been minimized.

Sign in to view
@virajjasani virajjasani requested a review from jojochuang July 15, 2021 18:08
@hadoop-yetus

This comment has been minimized.

Sign in to view
@virajjasani virajjasani requested a review from aajisaka July 16, 2021 14:04
@hadoop-yetus

This comment has been minimized.

Sign in to view
aajisaka
aajisaka reviewed Jul 19, 2021
View reviewed changes
Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @virajjasani for addressing the comments. Now I have a few additional minor comments, and I'm +1 if these are addressed.

@hadoop-yetus

This comment has been minimized.

Sign in to view
@hadoop-yetus

This comment has been minimized.

Sign in to view
@virajjasani virajjasani requested a review from aajisaka July 19, 2021 16:47
@hadoop-yetus

This comment has been minimized.

Sign in to view
aajisaka
aajisaka approved these changes Jul 20, 2021
View reviewed changes
Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thank you.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 30m 51s trunk passed
+1 💚 compile 20m 51s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 18m 22s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 10s trunk passed
+1 💚 mvnsite 1m 36s trunk passed
+1 💚 javadoc 1m 8s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 39s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 30s trunk passed
+1 💚 shadedclient 16m 42s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 57s the patch passed
+1 💚 compile 21m 41s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 21m 41s the patch passed
+1 💚 compile 19m 37s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 19m 37s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 8s the patch passed
+1 💚 mvnsite 1m 42s the patch passed
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 javadoc 1m 2s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 36s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 39s the patch passed
+1 💚 shadedclient 17m 1s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 57s hadoop-common in the patch passed.
+1 💚 asflicense 0m 49s The patch does not generate ASF License warnings.
180m 19s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3198/14/artifact/out/Dockerfile
GITHUB PR #3198
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname Linux 39bc92392198 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cee5775
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3198/14/testReport/
Max. process+thread count 3031 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3198/14/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@virajjasani
Copy link
Contributor Author

QA results look good now. (earlier one had some temporary issue with hadoop-yarn-ui module, hence build was failing)

@aajisaka
Copy link
Member

@jojochuang Would you review this?

jojochuang
jojochuang approved these changes Jul 20, 2021
View reviewed changes
Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 30m 49s trunk passed
+1 💚 compile 23m 24s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 19m 25s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 7s trunk passed
+1 💚 mvnsite 1m 34s trunk passed
+1 💚 javadoc 1m 6s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 38s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 25s trunk passed
+1 💚 shadedclient 16m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 0s the patch passed
+1 💚 compile 20m 21s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 20m 21s the patch passed
+1 💚 compile 18m 8s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 18m 8s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 10s the patch passed
+1 💚 mvnsite 1m 33s the patch passed
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 1m 6s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 40s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 35s the patch passed
+1 💚 shadedclient 15m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 17m 13s hadoop-common in the patch passed.
+1 💚 asflicense 0m 58s The patch does not generate ASF License warnings.
180m 38s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3198/15/artifact/out/Dockerfile
GITHUB PR #3198
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname Linux a120ec647893 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cee5775
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3198/15/testReport/
Max. process+thread count 1251 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3198/15/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@jojochuang jojochuang merged commit e1d00ad into apache:trunk Jul 20, 2021
asfgit pushed a commit that referenced this pull request Jul 20, 2021
@virajjasani @jojochuang
HADOOP-16290. Enable RpcMetrics units to be configurable (#3198)
ec33119 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e1d00ad)
@virajjasani virajjasani deleted the HADOOP-16290-trunk branch July 20, 2021 07:26
@aajisaka
Copy link
Member

Thank you @virajjasani and @jojochuang!

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021
@virajjasani
HADOOP-16290. Enable RpcMetrics units to be configurable (apache#3198)
adbe2f1 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jojochuang jojochuang jojochuang approved these changes

@aajisaka aajisaka aajisaka approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@virajjasani @hadoop-yetus @aajisaka @jojochuang