HDFS-14989. Add a 'swapBlockList' operation to Namenode. #1756

avijayanhwx · 2019-12-11T20:24:00Z

This is the first patch HDFS-14989 for the In Place Erasure Coding Feature HDFS-14978.

Work Done
The swapBlockList takes two parameters, a source file and a destination file. This operation swaps the blocks belonging to the source and the destination atomically.

The namespace metadata of interest is the INodeFile class. A file (INodeFile) contains a header composed of PREFERRED_BLOCK_SIZE, BLOCK_LAYOUT_AND_REDUNDANCY and STORAGE_POLICY_ID. In addition, an INodeFile contains a list of blocks (BlockInfo[]). The operation will swap BLOCK_LAYOUT_AND_REDUNDANCY header bits and the block lists. But it will not touch other fields. To avoid complication, this operation will abort if either file is open (isUnderConstruction() == true)

Note : This patch is intentionally not exposing the client side API to use this operation. That will be handled in a separate JIRA where the client side API will involve using this operation to carry out EC conversion.

Added unit tests for the operation as well.

avijayanhwx · 2019-12-11T20:24:42Z

@jojochuang / @arp7 Please review when you have a chance.

hadoop-yetus · 2019-12-12T00:36:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 54s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 22s	Maven dependency ordering for branch
+1 💚	mvninstall	23m 59s	trunk passed
+1 💚	compile	20m 6s	trunk passed
+1 💚	checkstyle	2m 54s	trunk passed
+1 💚	mvnsite	2m 41s	trunk passed
+1 💚	shadedclient	21m 1s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 2s	trunk passed
+0 🆗	spotbugs	3m 28s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	3m 26s	hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant findbugs warnings.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 59s	the patch passed
+1 💚	compile	18m 8s	the patch passed
-1 ❌	javac	18m 8s	root generated 5 new + 1863 unchanged - 5 fixed = 1868 total (was 1868)
-0 ⚠️	checkstyle	2m 57s	root: The patch generated 9 new + 264 unchanged - 0 fixed = 273 total (was 264)
+1 💚	mvnsite	2m 59s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 4s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 4s	the patch passed
+1 💚	findbugs	6m 39s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 19s	hadoop-common in the patch passed.
-1 ❌	unit	108m 33s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 53s	The patch does not generate ASF License warnings.
		251m 1s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSClientRetries
	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/1/artifact/out/Dockerfile
GITHUB PR	#1756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2b672f2db5d6 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / b56c08b
Default Java	1.8.0_222
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/1/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/1/artifact/out/diff-compile-javac-root.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/1/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/1/testReport/
Max. process+thread count	2662 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

…yle issues)

hadoop-yetus · 2019-12-12T22:44:45Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 20s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	24m 3s	trunk passed
+1 💚	compile	23m 55s	trunk passed
+1 💚	checkstyle	3m 22s	trunk passed
+1 💚	mvnsite	3m 16s	trunk passed
+1 💚	shadedclient	24m 20s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 22s	trunk passed
+0 🆗	spotbugs	3m 55s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	3m 54s	hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant findbugs warnings.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 26s	the patch passed
+1 💚	compile	22m 33s	the patch passed
+1 💚	javac	22m 33s	the patch passed
+1 💚	checkstyle	3m 24s	the patch passed
+1 💚	mvnsite	3m 13s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 40s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 56s	the patch passed
+1 💚	findbugs	6m 18s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 43s	hadoop-common in the patch passed.
-1 ❌	unit	111m 10s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		267m 22s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.TestDeadNodeDetection
	hadoop.hdfs.TestDFSInputStream

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/2/artifact/out/Dockerfile
GITHUB PR	#1756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 243580be308c 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / c210ced
Default Java	1.8.0_222
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/2/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/2/testReport/
Max. process+thread count	2916 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

arp7 · 2019-12-18T17:43:48Z

Hi @avijayanhwx , thanks for putting up this improvement. I am still reviewing it, couple of high-level comments:

This will require a NameNodeLayoutVersion increase.
The swap operation should accept a generationStamp as input and fail if the generation stamp of the target file is different. This prevents race condition wrt append/truncate.

...s-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

...hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java

arp7 · 2019-12-18T18:40:04Z

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

+        && Arrays.asList(options).contains(
+            Options.SwapBlockList.ONE_WAY_BLOCK_SWAP);
+    if (!overwrite) {
+      srcINodeFile.replaceBlocks(dstINodeFileBlocks);


Also update mtime for srcINodeFile here?

@arp7 The srcInodeFile's mtime is modified in Line 117.

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

arp7

The approach look good to me overall, the code was very easy to follow. Left a few high level and a few specific comments.

arp7 · 2019-12-18T18:55:20Z

Also we need to update the BlockCollection back-reference in each affected block.

avijayanhwx

Thanks for the review @arp7. I will address them and post a new patch.

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

avijayanhwx · 2019-12-18T18:58:11Z

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

+        && Arrays.asList(options).contains(
+            Options.SwapBlockList.ONE_WAY_BLOCK_SWAP);
+    if (!overwrite) {
+      srcINodeFile.replaceBlocks(dstINodeFileBlocks);


@arp7 The srcInodeFile's mtime is modified in Line 117.

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

...hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java

...s-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java

hadoop-yetus · 2019-12-19T03:03:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 20s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 23s	Maven dependency ordering for branch
+1 💚	mvninstall	22m 34s	trunk passed
+1 💚	compile	19m 25s	trunk passed
+1 💚	checkstyle	3m 2s	trunk passed
+1 💚	mvnsite	2m 43s	trunk passed
+1 💚	shadedclient	21m 2s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 4s	trunk passed
+0 🆗	spotbugs	3m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 19s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 59s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 55s	the patch passed
+1 💚	compile	18m 16s	the patch passed
+1 💚	javac	18m 16s	the patch passed
+1 💚	checkstyle	3m 4s	the patch passed
+1 💚	mvnsite	2m 41s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 11s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 57s	the patch passed
+1 💚	findbugs	5m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 21s	hadoop-common in the patch passed.
-1 ❌	unit	106m 15s	hadoop-hdfs in the patch failed.
-1 ❌	asflicense	0m 55s	The patch generated 2 ASF License warnings.
		244m 48s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
	hadoop.cli.TestCacheAdminCLI
	hadoop.cli.TestErasureCodingCLI
	hadoop.cli.TestHDFSCLI
	hadoop.tracing.TestTracingShortCircuitLocalRead
	hadoop.tracing.TestTraceAdmin

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/3/artifact/out/Dockerfile
GITHUB PR	#1756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 15fa4722ff18 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `7b93575`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/3/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/3/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	2777 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-01-05T04:01:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 17s	Maven dependency ordering for branch
+1 💚	mvninstall	19m 17s	trunk passed
+1 💚	compile	18m 36s	trunk passed
+1 💚	checkstyle	2m 41s	trunk passed
+1 💚	mvnsite	2m 59s	trunk passed
+1 💚	shadedclient	19m 50s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 9s	trunk passed
+0 🆗	spotbugs	3m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 12s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 0s	the patch passed
+1 💚	compile	17m 54s	the patch passed
-1 ❌	javac	17m 54s	root generated 5 new + 1862 unchanged - 5 fixed = 1867 total (was 1867)
-0 ⚠️	checkstyle	2m 41s	root: The patch generated 1 new + 257 unchanged - 0 fixed = 258 total (was 257)
+1 💚	mvnsite	3m 3s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	12m 56s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 19s	the patch passed
+1 💚	findbugs	5m 51s	the patch passed
		_ Other Tests _
+1 💚	unit	10m 20s	hadoop-common in the patch passed.
-1 ❌	unit	107m 21s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 0s	The patch does not generate ASF License warnings.
		240m 2s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.TestDatanodeDeath
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
	hadoop.hdfs.server.mover.TestMover
	hadoop.hdfs.TestClientProtocolForPipelineRecovery
	hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
	hadoop.hdfs.TestDistributedFileSystem
	hadoop.hdfs.tools.TestDFSAdminWithHA
	hadoop.hdfs.TestMissingBlocksAlert
	hadoop.hdfs.TestErasureCodingExerciseAPIs
	hadoop.hdfs.tools.TestECAdmin
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.TestErasureCodingPolicies
	hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
	hadoop.hdfs.server.namenode.TestRedudantBlocks
	hadoop.hdfs.TestWriteReadStripedFile
	hadoop.hdfs.TestReadStripedFileWithDNFailure
	hadoop.hdfs.TestEncryptionZones
	hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy
	hadoop.hdfs.TestDFSStripedInputStream
	hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover
	hadoop.hdfs.TestBlockStoragePolicy
	hadoop.hdfs.TestDecommission
	hadoop.hdfs.TestMultipleNNPortQOP
	hadoop.hdfs.TestDecommissionWithStriped
	hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
	hadoop.hdfs.TestSetrepDecreasing
	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.TestErasureCodingPolicyWithSnapshot

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/4/artifact/out/Dockerfile
GITHUB PR	#1756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0b0491fa6c0b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f8644fb`
Default Java	1.8.0_232
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/4/artifact/out/diff-compile-javac-root.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/4/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/4/testReport/
Max. process+thread count	3868 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-01-05T04:06:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 3s	Maven dependency ordering for branch
+1 💚	mvninstall	19m 9s	trunk passed
+1 💚	compile	18m 39s	trunk passed
+1 💚	checkstyle	2m 43s	trunk passed
+1 💚	mvnsite	3m 1s	trunk passed
+1 💚	shadedclient	19m 39s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 13s	trunk passed
+0 🆗	spotbugs	3m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 19s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 1s	the patch passed
+1 💚	compile	18m 0s	the patch passed
+1 💚	javac	18m 0s	the patch passed
-0 ⚠️	checkstyle	2m 45s	root: The patch generated 1 new + 257 unchanged - 0 fixed = 258 total (was 257)
+1 💚	mvnsite	2m 58s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	12m 56s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 16s	the patch passed
+1 💚	findbugs	5m 51s	the patch passed
		_ Other Tests _
+1 💚	unit	10m 1s	hadoop-common in the patch passed.
-1 ❌	unit	113m 12s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 6s	The patch does not generate ASF License warnings.
		245m 26s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStripedOutputStream
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
	hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
	hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
	hadoop.hdfs.TestRollingUpgradeRollback
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.server.blockmanagement.TestBlockManager
	hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
	hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.TestErasureCodingExerciseAPIs
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
	hadoop.hdfs.TestErasureCodingPolicies
	hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
	hadoop.hdfs.server.namenode.TestRedudantBlocks
	hadoop.hdfs.server.blockmanagement.TestSequentialBlockId
	hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped
	hadoop.hdfs.TestParallelUnixDomainRead
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.TestEncryptionZones
	hadoop.hdfs.TestFileAppend2
	hadoop.hdfs.TestDistributedFileSystemWithECFile
	hadoop.hdfs.server.namenode.TestFSEditLogLoader
	hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
	hadoop.hdfs.server.namenode.TestAddStripedBlocks
	hadoop.hdfs.TestFileCreationClient
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/5/artifact/out/Dockerfile
GITHUB PR	#1756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 62babd16f716 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f8644fb`
Default Java	1.8.0_232
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/5/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/5/testReport/
Max. process+thread count	4778 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-01-05T08:11:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for branch
+1 💚	mvninstall	18m 10s	trunk passed
+1 💚	compile	16m 41s	trunk passed
+1 💚	checkstyle	2m 42s	trunk passed
+1 💚	mvnsite	2m 49s	trunk passed
+1 💚	shadedclient	19m 15s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 0s	trunk passed
+0 🆗	spotbugs	2m 58s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	4m 58s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 53s	the patch passed
+1 💚	compile	15m 51s	the patch passed
-1 ❌	javac	15m 51s	root generated 5 new + 1862 unchanged - 5 fixed = 1867 total (was 1867)
+1 💚	checkstyle	2m 41s	the patch passed
+1 💚	mvnsite	2m 48s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	12m 41s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 5s	the patch passed
+1 💚	findbugs	5m 16s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 18s	hadoop-common in the patch passed.
-1 ❌	unit	95m 8s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 3s	The patch does not generate ASF License warnings.
		218m 24s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
	hadoop.hdfs.server.namenode.TestRedudantBlocks
	hadoop.hdfs.TestDecommissionWithStriped
	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.TestDeadNodeDetection
	hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/6/artifact/out/Dockerfile
GITHUB PR	#1756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 6d15ceb9d3ca 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f8644fb`
Default Java	1.8.0_232
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/6/artifact/out/diff-compile-javac-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/6/testReport/
Max. process+thread count	5158 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1756/6/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

jojochuang

I think the PR is good to go, but please wait for the feature branch to be created.

avijayanhwx · 2020-01-27T05:37:08Z

Closing this one. Will create another PR against feature branch.

HDFS-14989. Add a 'swapBlockList' operation to Namenode.

5e430d5

HDFS-14989. Add a 'swapBlockList' operation to Namenode. (Fix checkst…

aba3317

…yle issues)

arp7 self-requested a review December 17, 2019 20:29