HDFS-15749. Make size of editPendingQ can be configurable #2572

maobaolong · 2020-12-24T00:28:10Z

https://issues.apache.org/jira/browse/HDFS-15749

The hardcode 4096 maybe too small for our scenario, so I want to make it configurable, and default to 4096.

hadoop-yetus · 2020-12-24T00:30:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 4s		#2572 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#2572
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-12-24T03:43:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 59s		trunk passed
+1 💚	compile	1m 19s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	1m 15s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	shadedclient	17m 52s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 26s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	3m 7s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 5s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 11s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	1m 11s		the patch passed
+1 💚	compile	1m 5s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	1m 5s		the patch passed
-0 ⚠️	checkstyle	0m 51s	/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 464 unchanged - 0 fixed = 466 total (was 464)
+1 💚	mvnsite	1m 10s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 1s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 23s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 4s		the patch passed
			_ Other Tests _
-1 ❌	unit	98m 24s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		188m 38s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/2/artifact/out/Dockerfile
GITHUB PR	#2572
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 9c9faadd5495 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `16a2050`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/2/testReport/
Max. process+thread count	3800 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

maobaolong · 2020-12-24T04:03:49Z

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestBalancer.testBalancerCliWithExcludeList:1462->doTest:808->doTest:939->runBalancerCli:1100->waitForBalancer:674
[ERROR]   TestBalancer.testBalancerCliWithExcludeListInAFile:1494->doTest:808->doTest:939->runBalancerCli:1100->waitForBalancer:674
[ERROR]   TestBalancer.testBalancerCliWithIncludeListInAFile:1584->doTest:808->doTest:939->runBalancerCli:1100->waitForBalancer:679
[ERROR]   TestBalancer.testBalancerWithExcludeList:1431->doTest:808->doTest:941->runBalancer:978->waitForBalancer:674
[ERROR]   TestBalancer.testBalancerWithIncludeList:1524->doTest:808->doTest:941->runBalancer:978->waitForBalancer:679
[ERROR]   TestBalancer.testBalancerWithIncludeListWithPorts:1539->doTest:808->doTest:941->runBalancer:978->waitForBalancer:679
[ERROR]   TestBalancer.testBalancerWithSortTopNodes:2312 expected:<900> but was:<950>
[ERROR] Errors: 
[ERROR]   TestDFSInotifyEventInputStreamKerberized.testWithKerberizedCluster:116 ? IO De...
[ERROR]   TestBalancer.testBalancerWithStripedFile:2068->doTestBalancerWithStripedFile:2121->runBalancer:975->waitForHeartBeat:464 Timeout
[ERROR]   TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks:80 ? TestTimedOut
[INFO] 
[ERROR] Tests run: 6706, Failures: 7, Errors: 3, Skipped: 24

I think the failed tests are not related to this PR, so I just closed and reopen this PR to trigger another new CI checks.

ferhui · 2020-12-24T05:43:57Z

@maobaolong Thanks for report and fix.
It looks good, please fix the checkstyle that CI reports.

maobaolong · 2020-12-24T08:07:34Z

@ferhui Thanks for your review and suggestion, I fixed the style issue and PTAL.

hadoop-yetus · 2020-12-24T11:01:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
-1 ❌	mvninstall	5m 45s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	2m 13s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
-1 ❌	compile	0m 23s	/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt	hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.
+1 💚	checkstyle	1m 3s		trunk passed
+1 💚	mvnsite	1m 24s		trunk passed
+1 💚	shadedclient	20m 48s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 21s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	3m 3s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 1s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	1m 13s		the patch passed
+1 💚	compile	1m 5s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
-1 ❌	javac	1m 5s	/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt	hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~~18.04-b01 generated 583 new + 0 unchanged - 0 fixed = 583 total (was 0)
+1 💚	checkstyle	0m 53s		the patch passed
+1 💚	mvnsite	1m 15s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 10s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 24s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 4s		the patch passed
			_ Other Tests _
-1 ❌	unit	107m 28s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		173m 32s

Reason	Tests
Failed junit tests	hadoop.hdfs.security.TestDelegationTokenForProxyUser
	hadoop.hdfs.TestLeaseRecovery
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestFileChecksum

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/3/artifact/out/Dockerfile
GITHUB PR	#2572
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux a1c5935cbe14 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `16a2050`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/3/testReport/
Max. process+thread count	4258 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

maobaolong · 2020-12-24T12:07:19Z

@ferhui The CI checks failed, but I don't think it caused by my PR, PTAL. thank you.

hadoop-yetus · 2020-12-25T03:08:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 10s		trunk passed
+1 💚	compile	1m 19s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	1m 12s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	shadedclient	17m 30s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 28s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	3m 8s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 6s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 11s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	1m 11s		the patch passed
+1 💚	compile	1m 5s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	1m 5s		the patch passed
+1 💚	checkstyle	0m 50s		the patch passed
+1 💚	mvnsite	1m 10s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	shadedclient	14m 47s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 21s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 2s		the patch passed
			_ Other Tests _
-1 ❌	unit	97m 12s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		186m 41s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/4/artifact/out/Dockerfile
GITHUB PR	#2572
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux fe777561c5a0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `005b854`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/4/testReport/
Max. process+thread count	4627 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2572/4/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ferhui · 2020-12-25T03:34:10Z

+1

maobaolong · 2020-12-25T04:41:42Z

@ferhui Thank you for your review and approved.

ferhui · 2020-12-28T06:20:54Z

@maobaolong Thanks for contribution, merged

HDFS-15749. Make size of editPendingQ can be configurable

a46ff28

maobaolong force-pushed the configurable-editPQ branch from b0769a5 to a46ff28 Compare December 24, 2020 00:33