HDFS-14350:dfs.datanode.ec.reconstruction.threads not take effect #582

hunshenshi · 2019-03-09T05:42:31Z

In ErasureCodingWorker, stripedReconstructionPool is create by

initializeStripedBlkReconstructionThreadPool(conf.getInt(
    DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_KEY,
    DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_DEFAULT));

private void initializeStripedBlkReconstructionThreadPool(int numThreads) {
  LOG.debug("Using striped block reconstruction; pool threads={}",
      numThreads);
  stripedReconstructionPool = DFSUtilClient.getThreadPoolExecutor(2,
      numThreads, 60, new LinkedBlockingQueue<>(),
      "StripedBlockReconstruction-", false);
  stripedReconstructionPool.allowCoreThreadTimeOut(true);
}

so stripedReconstructionPool is a ThreadPoolExecutor, and the queue is a LinkedBlockingQueue, then the active thread is awalys 2, the dfs.datanode.ec.reconstruction.threads not take effect.

update

hadoop-yetus · 2019-03-09T07:55:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	27	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1017	trunk passed
+1	compile	61	trunk passed
+1	checkstyle	53	trunk passed
+1	mvnsite	66	trunk passed
+1	shadedclient	767	branch has no errors when building and testing our client artifacts.
+1	findbugs	118	trunk passed
+1	javadoc	48	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	56	the patch passed
+1	compile	52	the patch passed
+1	javac	52	the patch passed
+1	checkstyle	42	the patch passed
+1	mvnsite	56	the patch passed
+1	whitespace	1	The patch has no whitespace issues.
+1	shadedclient	673	patch has no errors when building and testing our client artifacts.
+1	findbugs	125	the patch passed
+1	javadoc	48	the patch passed
		_ Other Tests _
-1	unit	4591	hadoop-hdfs in the patch failed.
+1	asflicense	38	The patch does not generate ASF License warnings.
		7886

Reason	Tests
Failed junit tests	hadoop.hdfs.web.TestWebHdfsTimeouts

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/1/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ac62574b9078 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c072458`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
findbugs	v3.1.0-RC1
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/1/testReport/
Max. process+thread count	4795 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/1/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-07-19T16:59:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	92	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1240	trunk passed
+1	compile	64	trunk passed
+1	checkstyle	41	trunk passed
+1	mvnsite	67	trunk passed
+1	shadedclient	821	branch has no errors when building and testing our client artifacts.
+1	javadoc	56	trunk passed
0	spotbugs	190	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	188	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	70	the patch passed
+1	compile	63	the patch passed
+1	javac	63	the patch passed
+1	checkstyle	39	the patch passed
+1	mvnsite	66	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	778	patch has no errors when building and testing our client artifacts.
+1	javadoc	54	the patch passed
+1	findbugs	202	the patch passed
		_ Other Tests _
-1	unit	6634	hadoop-hdfs in the patch failed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		10582

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
	hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
	hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
	hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/2/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 4f713509fdb7 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `cd967c7`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/2/testReport/
Max. process+thread count	2861 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-07-26T17:57:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	85	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1112	trunk passed
+1	compile	64	trunk passed
+1	checkstyle	41	trunk passed
+1	mvnsite	69	trunk passed
+1	shadedclient	762	branch has no errors when building and testing our client artifacts.
-1	javadoc	24	hadoop-hdfs in trunk failed.
0	spotbugs	166	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	164	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	65	the patch passed
+1	compile	55	the patch passed
+1	javac	55	the patch passed
+1	checkstyle	36	the patch passed
+1	mvnsite	62	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	678	patch has no errors when building and testing our client artifacts.
+1	javadoc	50	the patch passed
+1	findbugs	169	the patch passed
		_ Other Tests _
-1	unit	6382	hadoop-hdfs in the patch failed.
-1	asflicense	33	The patch generated 1 ASF License warnings.
		9894

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/3/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 6ff9f36636f3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c7c7a88`
Default Java	1.8.0_212
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/3/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/3/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	4578 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-02T04:45:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	78	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1209	trunk passed
+1	compile	71	trunk passed
+1	checkstyle	50	trunk passed
+1	mvnsite	81	trunk passed
+1	shadedclient	903	branch has no errors when building and testing our client artifacts.
+1	javadoc	52	trunk passed
0	spotbugs	188	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	185	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	75	the patch passed
+1	compile	66	the patch passed
+1	javac	66	the patch passed
+1	checkstyle	39	the patch passed
+1	mvnsite	70	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	807	patch has no errors when building and testing our client artifacts.
+1	javadoc	51	the patch passed
+1	findbugs	182	the patch passed
		_ Other Tests _
-1	unit	6367	hadoop-hdfs in the patch failed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		10365

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.tools.TestDFSZKFailoverController
	hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
	hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
	hadoop.hdfs.server.namenode.ha.TestHASafeMode
	hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
	hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/4/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux d40cb95b3f21 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / e20b195
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/4/testReport/
Max. process+thread count	3634 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-08T06:22:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	52	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1073	trunk passed
+1	compile	57	trunk passed
+1	checkstyle	42	trunk passed
+1	mvnsite	65	trunk passed
+1	shadedclient	732	branch has no errors when building and testing our client artifacts.
+1	javadoc	48	trunk passed
0	spotbugs	170	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	167	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	60	the patch passed
+1	compile	55	the patch passed
+1	javac	55	the patch passed
+1	checkstyle	36	the patch passed
+1	mvnsite	63	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	721	patch has no errors when building and testing our client artifacts.
+1	javadoc	52	the patch passed
+1	findbugs	169	the patch passed
		_ Other Tests _
-1	unit	4974	hadoop-hdfs in the patch failed.
-1	asflicense	31	The patch generated 4 ASF License warnings.
		8479

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/5/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 007c7e0882a6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b0131bc`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/5/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/5/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	4338 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-16T20:21:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1173	trunk passed
+1	compile	52	trunk passed
+1	checkstyle	38	trunk passed
+1	mvnsite	58	trunk passed
+1	shadedclient	722	branch has no errors when building and testing our client artifacts.
+1	javadoc	48	trunk passed
0	spotbugs	159	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	157	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	56	the patch passed
+1	compile	49	the patch passed
+1	javac	49	the patch passed
+1	checkstyle	34	the patch passed
+1	mvnsite	56	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	685	patch has no errors when building and testing our client artifacts.
+1	javadoc	45	the patch passed
+1	findbugs	176	the patch passed
		_ Other Tests _
-1	unit	5151	hadoop-hdfs in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		8650

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
	hadoop.hdfs.server.namenode.TestNamenodeCapacityReport

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/6/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2f8024efd289 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e356e4f`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/6/testReport/
Max. process+thread count	4151 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/6/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

jojochuang · 2019-08-17T00:15:09Z

I am not sure this is the right fix -- reading at the code, clients keeps 2 threads even during idle, and dfs.datanode.ec.reconstruction.threads specifies the max number of threads. That looks reasonable.
Perhaps all we should do is to update the description of the configuration key in hdfs-default.xml.

hunshenshi · 2019-08-18T03:23:29Z

hi, @jojochuang Thanks for review.
The queue of ThreadPoolExecutor is a LinkedBlockingQueue, the queue will not be filled, so the max num will not take effect. So I think the active thread is awalys 2, dfs.datanode.ec.reconstruction.threads not take effect.

hadoop-yetus · 2019-08-20T17:42:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	33	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1222	trunk passed
+1	compile	64	trunk passed
+1	checkstyle	46	trunk passed
+1	mvnsite	71	trunk passed
+1	shadedclient	845	branch has no errors when building and testing our client artifacts.
+1	javadoc	53	trunk passed
0	spotbugs	174	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	171	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	65	the patch passed
+1	compile	57	the patch passed
+1	javac	57	the patch passed
+1	checkstyle	42	the patch passed
+1	mvnsite	69	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	809	patch has no errors when building and testing our client artifacts.
+1	javadoc	53	the patch passed
+1	findbugs	183	the patch passed
		_ Other Tests _
-1	unit	5277	hadoop-hdfs in the patch failed.
-1	asflicense	37	The patch generated 1 ASF License warnings.
		9149

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
	hadoop.hdfs.server.namenode.ha.TestHASafeMode
	hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.tools.TestDFSZKFailoverController
	hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/7/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e0a6ef064c5d 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `094d736`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/7/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/7/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	3364 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/7/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-22T16:54:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	46	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1047	trunk passed
+1	compile	58	trunk passed
+1	checkstyle	40	trunk passed
+1	mvnsite	62	trunk passed
+1	shadedclient	738	branch has no errors when building and testing our client artifacts.
+1	javadoc	47	trunk passed
0	spotbugs	162	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	160	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	61	the patch passed
+1	compile	53	the patch passed
+1	javac	53	the patch passed
+1	checkstyle	39	the patch passed
+1	mvnsite	59	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	675	patch has no errors when building and testing our client artifacts.
+1	javadoc	50	the patch passed
+1	findbugs	158	the patch passed
		_ Other Tests _
-1	unit	4839	hadoop-hdfs in the patch failed.
-1	asflicense	31	The patch generated 1 ASF License warnings.
		8210

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestSaveNamespace
	hadoop.hdfs.server.namenode.ha.TestHASafeMode
	hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
	hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/8/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 26c166f75554 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `69ddb36`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/8/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/8/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	4421 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/8/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-27T13:38:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	38	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1030	trunk passed
+1	compile	65	trunk passed
+1	checkstyle	45	trunk passed
+1	mvnsite	70	trunk passed
+1	shadedclient	783	branch has no errors when building and testing our client artifacts.
+1	javadoc	54	trunk passed
0	spotbugs	159	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	156	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	62	the patch passed
+1	compile	57	the patch passed
+1	javac	57	the patch passed
+1	checkstyle	40	the patch passed
+1	mvnsite	65	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	728	patch has no errors when building and testing our client artifacts.
+1	javadoc	54	the patch passed
+1	findbugs	162	the patch passed
		_ Other Tests _
-1	unit	5070	hadoop-hdfs in the patch failed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		8590

Reason	Tests
Failed junit tests	hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
	hadoop.hdfs.server.namenode.TestAddStripedBlocks
	hadoop.hdfs.TestErasureCodingExerciseAPIs
	hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/9/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c49c2b7bea1a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3329257`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/9/testReport/
Max. process+thread count	5101 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/9/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-03T05:34:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	52	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1293	trunk passed
+1	compile	68	trunk passed
+1	checkstyle	49	trunk passed
+1	mvnsite	78	trunk passed
+1	shadedclient	970	branch has no errors when building and testing our client artifacts.
+1	javadoc	63	trunk passed
0	spotbugs	169	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	167	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	65	the patch passed
+1	compile	63	the patch passed
+1	javac	63	the patch passed
+1	checkstyle	40	the patch passed
+1	mvnsite	68	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	739	patch has no errors when building and testing our client artifacts.
+1	javadoc	54	the patch passed
+1	findbugs	171	the patch passed
		_ Other Tests _
-1	unit	5279	hadoop-hdfs in the patch failed.
+1	asflicense	37	The patch does not generate ASF License warnings.
		9324

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
	hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
	hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
	hadoop.hdfs.TestMultipleNNPortQOP
	hadoop.hdfs.server.datanode.TestBPOfferService

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-582/10/artifact/out/Dockerfile
GITHUB PR	#582
JIRA Issue	HDFS-14350
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b4895b1fe5a4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `915cbc9`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/10/testReport/
Max. process+thread count	4181 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-582/10/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Handle end of stream envelopes in bootstrap chooser and don't invoke check offsets since offset of end of stream are not comparable. Author: Bharath Kumarasubramanian <bkumaras@linkedin.com> Reviewers: Xinyu Liu <xinyu@apache.org> Closes apache#582 from bharathkk/end-of-stream

hadoop-yetus · 2020-07-31T19:40:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	patch	0m 5s	#582 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#582
JIRA Issue	HDFS-14350
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-582/3/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-07-31T21:01:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	patch	0m 4s	#582 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#582
JIRA Issue	HDFS-14350
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-582/4/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-09-09T18:02:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	patch	0m 6s	#582 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#582
JIRA Issue	HDFS-14350
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-582/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-07-24T06:08:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 18s		#582 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#582
JIRA Issue	HDFS-14350
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-582/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hunshenshi added 2 commits March 9, 2019 13:29

Merge pull request #1 from apache/trunk

5c13222

update

HDFS-14350:dfs.datanode.ec.reconstruction.threads not take effect

592ec2d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-14350:dfs.datanode.ec.reconstruction.threads not take effect #582

HDFS-14350:dfs.datanode.ec.reconstruction.threads not take effect #582

hunshenshi commented Mar 9, 2019

hadoop-yetus commented Mar 9, 2019

hadoop-yetus commented Jul 19, 2019

hadoop-yetus commented Jul 26, 2019

hadoop-yetus commented Aug 2, 2019

hadoop-yetus commented Aug 8, 2019

hadoop-yetus commented Aug 16, 2019

jojochuang commented Aug 17, 2019

hunshenshi commented Aug 18, 2019

hadoop-yetus commented Aug 20, 2019

hadoop-yetus commented Aug 22, 2019

hadoop-yetus commented Aug 27, 2019

hadoop-yetus commented Sep 3, 2019

hadoop-yetus commented Jul 31, 2020

hadoop-yetus commented Jul 31, 2020

hadoop-yetus commented Sep 9, 2020

hadoop-yetus commented Jul 24, 2021

HDFS-14350:dfs.datanode.ec.reconstruction.threads not take effect #582

Are you sure you want to change the base?

HDFS-14350:dfs.datanode.ec.reconstruction.threads not take effect #582

Conversation

hunshenshi commented Mar 9, 2019

hadoop-yetus commented Mar 9, 2019

hadoop-yetus commented Jul 19, 2019

hadoop-yetus commented Jul 26, 2019

hadoop-yetus commented Aug 2, 2019

hadoop-yetus commented Aug 8, 2019

hadoop-yetus commented Aug 16, 2019

jojochuang commented Aug 17, 2019

hunshenshi commented Aug 18, 2019

hadoop-yetus commented Aug 20, 2019

hadoop-yetus commented Aug 22, 2019

hadoop-yetus commented Aug 27, 2019

hadoop-yetus commented Sep 3, 2019

hadoop-yetus commented Jul 31, 2020

hadoop-yetus commented Jul 31, 2020

hadoop-yetus commented Sep 9, 2020

hadoop-yetus commented Jul 24, 2021