HDFS-16703. Enable RPC Timeout for some protocols of NameNode. #4660

ZanderXu · 2022-07-30T03:10:35Z

Description of PR

When I read some code about protocol, I found that only ClientNamenodeProtocolPB proxy with RPC timeout, other protocolPB proxy without RPC timeout, such as RefreshAuthorizationPolicyProtocolPB, RefreshUserMappingsProtocolPB, RefreshCallQueueProtocolPB, GetUserMappingsProtocolPB and NamenodeProtocolPB.

If proxy without rpc timeout, it will be blocked for a long time if the NN machine crash or bad network during writing or reading with NN.

So I feel that we should enable RPC timeout for all ProtocolPB.

hadoop-yetus · 2022-07-30T11:27:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 50s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 55s		trunk passed
+1 💚	compile	6m 49s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	6m 20s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 33s		trunk passed
+1 💚	mvnsite	2m 49s		trunk passed
+1 💚	javadoc	2m 12s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 32s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 35s		trunk passed
+1 💚	shadedclient	26m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 17s		the patch passed
+1 💚	compile	6m 37s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	6m 37s		hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135)
+1 💚	compile	6m 17s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 17s		hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 16s		the patch passed
+1 💚	mvnsite	2m 22s		the patch passed
+1 💚	javadoc	1m 42s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 8s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 20s		the patch passed
+1 💚	shadedclient	25m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 31s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	339m 19s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		495m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/1/artifact/out/Dockerfile
GITHUB PR	#4660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5ca4b2229ba9 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 03e1c7d8aab9f1f76e5a1ef7fd637fd6808fa991
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/1/testReport/
Max. process+thread count	2282 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2022-07-30T12:46:08Z

@ZanderXu I feel that this change is a bit risky, will this lead to instability of the service? Sometimes it is reasonable to configure without timeout. The main question is is it reasonable to use the same timeout configuration for different protocols?

ZanderXu · 2022-07-30T13:20:13Z

@slfan1989 Thanks for your review.

I feel that this change is a bit risky, will this lead to instability of the service? Sometimes it is reasonable to configure without timeout

Sorry, I didn't get your idea. Can you share more detailed information or cases?

The main question is is it reasonable to use the same timeout configuration for different protocols?

According to my practical experience, using one configuration ipc.client.rpc-timeout.ms is enough. Of course, if some masters feel it is necessary to distinguish this configuration, I will modify this patch.

slfan1989 · 2022-07-30T13:43:16Z

Sorry, I didn't get your idea. Can you share more detailed information or cases?

I mean you have increased the timeout configuration, will some protocols timeout frequently?

Frequent timeouts cause some operations to fail. I personally think that the performance of NN RPC is limited by NN FsSystemLock, which causes Rpc to not necessarily return at a regular time. Setting this may lead to a large number of RPC failures in some cases, resulting in overall instability.

let's see what other people think.

ZanderXu · 2022-07-30T13:56:35Z

Thanks @slfan1989 for your explain.

ipc.client.rpc-timeout.ms default is 2min. If client can‘t receive the response within 2min, maybe we should give a high priority to improving the performance of namenode.

If timeout is 0, Client will be blocked for a very long time, such as 15min, due to NN crash or bad network or other reasons.

slfan1989 · 2022-07-30T14:01:23Z

Thanks @slfan1989 for your explain.

ipc.client.rpc-timeout.ms default is 2min. If client can‘t receive the response within 2min, maybe we should give a high priority to improving the performance of namenode.

If timeout is 0, Client will be blocked for a very long time, such as 15min, due to NN crash or bad network or other reasons.

Thanks for the explanation, from my personal point of view, I don't really agree with this change, but I still listen to other partners' suggestions. I always feel that there is a reason for not setting timeout before.

ZanderXu · 2022-08-01T23:30:57Z

@goiri Sir, can you help me review this patch? Thanks
In our prod environment, we encountered the RBF NameNodeHeartbeatServices thread being blocked for a long time because one of the NameNode machine crashed. So I think this PR will also have an effect on RBF.

goiri · 2022-08-02T23:47:08Z

I tend to agree that we should avoid not changing the previous behavior.
Can we add one setting that allows keeping the old 0 for the default case and tune it for the rest?

ZanderXu · 2022-08-03T08:21:29Z

@goiri Thanks for your review.

Can we add one setting that allows keeping the old 0 for the default case and tune it for the rest?

Sure, I will modify it with different configuration for different protocol. cc @slfan1989

hadoop-yetus · 2022-08-05T11:30:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 57s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 11s		trunk passed
+1 💚	compile	6m 50s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	6m 29s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 34s		trunk passed
+1 💚	mvnsite	2m 49s		trunk passed
+1 💚	javadoc	2m 12s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 29s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 34s		trunk passed
+1 💚	shadedclient	25m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 18s		the patch passed
+1 💚	compile	6m 39s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	6m 39s		hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135)
+1 💚	compile	6m 15s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 15s		hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 16s		the patch passed
+1 💚	mvnsite	2m 24s		the patch passed
+1 💚	javadoc	1m 42s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 7s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 25s		the patch passed
+1 💚	shadedclient	26m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	345m 30s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		501m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/2/artifact/out/Dockerfile
GITHUB PR	#4660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux d70449d3d50f 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0f07908a8f36f476647123a5ffab1d6fe187ded6
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/2/testReport/
Max. process+thread count	2481 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java

ZanderXu · 2022-08-05T23:49:56Z

@goiri Master, thanks for your helping review.

slfan1989 · 2022-08-12T04:40:28Z

@ZanderXu Do we need to update the md file to explain what these parameters do?

ZanderXu · 2022-08-15T10:34:37Z

@slfan1989 Thanks for your review.

Do we need to update the md file to explain what these parameters do?

Do you mean which MD file? ipc.client.rpc-timeout.ms is not been explained. If you think we should do it, I will add some explanation for them.

hadoop-yetus · 2022-08-15T17:22:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 37s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 10s		trunk passed
+1 💚	compile	7m 4s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	6m 39s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 31s		trunk passed
+1 💚	mvnsite	2m 57s		trunk passed
+1 💚	javadoc	2m 14s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 46s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 53s		trunk passed
+1 💚	shadedclient	25m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 26s		the patch passed
+1 💚	compile	6m 45s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	6m 45s		hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135)
+1 💚	compile	6m 34s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 34s		hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 26s		the patch passed
+1 💚	mvnsite	2m 38s		the patch passed
+1 💚	javadoc	1m 52s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 17s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 47s		the patch passed
+1 💚	shadedclient	26m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 41s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	251m 46s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 16s		The patch does not generate ASF License warnings.
		408m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/3/artifact/out/Dockerfile
GITHUB PR	#4660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux aef4a00016af 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 07dd3584cd7543a033a6c510787242d977e08750
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/3/testReport/
Max. process+thread count	4031 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

goiri · 2022-08-15T20:45:01Z

@slfan1989 Thanks for your review.

Do we need to update the md file to explain what these parameters do?

Do you mean which MD file? ipc.client.rpc-timeout.ms is not been explained. If you think we should do it, I will add some explanation for them.

I'm not sure what is the best place to document.
TBH, I do not see a good place to document this; so it might be best to just have it in the XML.

ZanderXu · 2022-08-17T15:10:28Z

@slfan1989 Master, ping. Could you review it and make some suggestions or approval?

ZanderXu · 2022-08-22T23:19:24Z

@slfan1989 Master, ping.

slfan1989 · 2022-08-23T00:39:23Z

@ZanderXu Sorry for the late reply, the code looks fine. Thank you very much for your contribution.

slfan1989 · 2022-08-23T00:40:42Z

I am more worried about the use of these timeout configurations, because from my personal point of view, I don't know how to configure it.

Can we provide a guide for configuration?

Such As
ipc.rpc-timeout.for.refresh-user-mappings.ms, if this is not configured to 0, what is a better configuration value?

ZanderXu · 2022-08-23T02:08:56Z

@slfan1989 Master, thanks for your review.

ipc.rpc-timeout.for.refresh-user-mappings.ms, if this is not configured to 0, what is a better configuration value?

I have added some suggestion values in hdfs-default.xml, such as:

The default value of 0 indicates that timeout is disabled,
      which can be set to the same as ipc.client.rpc-timeout.ms, such as 120s.

What do you think of this?

hadoop-yetus · 2022-08-23T10:34:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 41s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 17s		trunk passed
+1 💚	compile	6m 59s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	6m 21s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 34s		trunk passed
+1 💚	mvnsite	2m 49s		trunk passed
+1 💚	javadoc	2m 12s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 29s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 35s		trunk passed
+1 💚	shadedclient	26m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 19s		the patch passed
+1 💚	compile	6m 37s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	6m 37s		hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135)
+1 💚	compile	6m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	6m 22s		hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 17s		the patch passed
+1 💚	mvnsite	2m 23s		the patch passed
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 7s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 21s		the patch passed
+1 💚	shadedclient	25m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	349m 13s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		506m 47s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.mover.TestMover
	hadoop.hdfs.server.namenode.ha.TestObserverNode

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/4/artifact/out/Dockerfile
GITHUB PR	#4660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 95644ecbe7f6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 281d70c8ce50542463a355826d81b07d0ce92fbc
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/4/testReport/
Max. process+thread count	2094 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ZanderXu · 2022-08-31T10:30:31Z

@slfan1989 Sir, can help take a look about this PR? It's helpful for hadoop admin.

ZanderXu · 2022-09-06T03:15:15Z

@slfan1989 Sir, sorry to ping you again. I still think this is a useful PR for Hadoop Admin. So I hope you can give me some more suggestions and push this PR forward. Thanks again.

hadoop-yetus · 2022-09-06T09:39:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 24s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 46s		trunk passed
+1 💚	compile	6m 6s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	5m 55s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 37s		trunk passed
+1 💚	mvnsite	2m 59s		trunk passed
+1 💚	javadoc	2m 14s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 36s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 27s		trunk passed
+1 💚	shadedclient	23m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 14s		the patch passed
+1 💚	compile	6m 1s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 1s		hadoop-hdfs-project-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135)
+1 💚	compile	5m 41s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	5m 41s		hadoop-hdfs-project-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 16s		the patch passed
+1 💚	mvnsite	2m 23s		the patch passed
+1 💚	javadoc	1m 39s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 8s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 1s		the patch passed
+1 💚	shadedclient	22m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 38s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	239m 8s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 8s		The patch does not generate ASF License warnings.
		385m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/5/artifact/out/Dockerfile
GITHUB PR	#4660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux ae65e76df820 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9967ad8`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/5/testReport/
Max. process+thread count	3798 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ZanderXu · 2022-09-15T11:11:08Z

@slfan1989 Sir, sorry to ping you again. I'm looking forward to your feedback. RPC Timeout is needed to prevent long-term blocking when the network is bad or machine crash. Thanks again.

ZanderXu · 2022-10-14T04:06:47Z

@slfan1989 Sir, about this PR, if you thinks it's difficult to configure it, how about just enabling a configurable timeout for NamenodeProtocolPB?

Because we encountered many times this problem in our prod environment that RBF can not sense the crashed namenode in time, because the NamenodeHeartbeatService are blocked to waiting for the response from namenode for a long time.

goiri reviewed Aug 5, 2022

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java Show resolved Hide resolved

goiri approved these changes Aug 5, 2022

View reviewed changes

ZanderXu force-pushed the HDFS-16703 branch from 0f07908 to 07dd358 Compare August 15, 2022 10:32

ZanderXu force-pushed the HDFS-16703 branch from 07dd358 to 281d70c Compare August 23, 2022 02:06

ZanderXu force-pushed the HDFS-16703 branch from 281d70c to 9967ad8 Compare September 6, 2022 03:11

HDFS-16703. Enable RPC Timeout for some protocols of NameNode.

1114822

ZanderXu force-pushed the HDFS-16703 branch from 9967ad8 to 1114822 Compare May 21, 2024 12:34

ZanderXu requested a review from haiyang1987 May 21, 2024 12:34

github-actions bot added HDFS trunk labels May 21, 2024

HDFS-16703. Enable RPC Timeout for some protocols of NameNode. #4660

Are you sure you want to change the base?

HDFS-16703. Enable RPC Timeout for some protocols of NameNode. #4660

Conversation

ZanderXu commented Jul 30, 2022

Description of PR

Uh oh!

hadoop-yetus commented Jul 30, 2022

Uh oh!

slfan1989 commented Jul 30, 2022

Uh oh!

ZanderXu commented Jul 30, 2022

Uh oh!

slfan1989 commented Jul 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZanderXu commented Jul 30, 2022

Uh oh!

slfan1989 commented Jul 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZanderXu commented Aug 1, 2022

Uh oh!

goiri commented Aug 2, 2022

Uh oh!

ZanderXu commented Aug 3, 2022

Uh oh!

hadoop-yetus commented Aug 5, 2022

Uh oh!

Uh oh!

ZanderXu commented Aug 5, 2022

Uh oh!

slfan1989 commented Aug 12, 2022

Uh oh!

ZanderXu commented Aug 15, 2022

Uh oh!

hadoop-yetus commented Aug 15, 2022

Uh oh!

goiri commented Aug 15, 2022

Uh oh!

ZanderXu commented Aug 17, 2022

Uh oh!

ZanderXu commented Aug 22, 2022

Uh oh!

slfan1989 commented Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slfan1989 commented Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZanderXu commented Aug 23, 2022

Uh oh!

hadoop-yetus commented Aug 23, 2022

Uh oh!

ZanderXu commented Aug 31, 2022

Uh oh!

ZanderXu commented Sep 6, 2022

Uh oh!

hadoop-yetus commented Sep 6, 2022

Uh oh!

ZanderXu commented Sep 15, 2022

Uh oh!

ZanderXu commented Oct 14, 2022

Uh oh!

Uh oh!

slfan1989 commented Jul 30, 2022 •

edited

Loading

slfan1989 commented Jul 30, 2022 •

edited

Loading

slfan1989 commented Aug 23, 2022 •

edited

Loading

slfan1989 commented Aug 23, 2022 •

edited

Loading