HDFS-14989. Add a 'swapBlockList' operation to Namenode. #1819

avijayanhwx · 2020-01-27T06:07:02Z

This is the first patch HDFS-14989 for the In Place Erasure Coding Feature HDFS-14978.

Work Done
The swapBlockList takes two parameters, a source file and a destination file. This operation swaps the blocks belonging to the source and the destination atomically.

The namespace metadata of interest is the INodeFile class. A file (INodeFile) contains a header composed of PREFERRED_BLOCK_SIZE, BLOCK_LAYOUT_AND_REDUNDANCY and STORAGE_POLICY_ID. In addition, an INodeFile contains a list of blocks (BlockInfo[]). The operation will swap BLOCK_LAYOUT_AND_REDUNDANCY header bits and the block lists. But it will not touch other fields. To avoid complication, this operation will abort if either file is open (isUnderConstruction() == true)

Note : This patch is intentionally not exposing the client side API to use this operation. That will be handled in a separate JIRA where the client side API will involve using this operation to carry out EC conversion.

Added unit tests for the operation as well.

…yle issues)

hadoop-yetus · 2020-01-27T10:46:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	30m 47s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ HDFS-14978_ec_conversion Compile Tests _
+0 🆗	mvndep	1m 8s	Maven dependency ordering for branch
+1 💚	mvninstall	21m 5s	HDFS-14978_ec_conversion passed
+1 💚	compile	17m 45s	HDFS-14978_ec_conversion passed
+1 💚	checkstyle	2m 55s	HDFS-14978_ec_conversion passed
+1 💚	mvnsite	2m 43s	HDFS-14978_ec_conversion passed
+1 💚	shadedclient	20m 48s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 55s	HDFS-14978_ec_conversion passed
+0 🆗	spotbugs	3m 6s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 8s	HDFS-14978_ec_conversion passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 54s	the patch passed
+1 💚	compile	17m 3s	the patch passed
+1 💚	javac	17m 3s	the patch passed
+1 💚	checkstyle	2m 51s	the patch passed
+1 💚	mvnsite	2m 42s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 0s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 1s	the patch passed
+1 💚	findbugs	5m 39s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 31s	hadoop-common in the patch passed.
-1 ❌	unit	116m 15s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		278m 35s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDeadNodeDetection

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/1/artifact/out/Dockerfile
GITHUB PR	#1819
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 05912709de22 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	HDFS-14978_ec_conversion / `839e607`
Default Java	1.8.0_232
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/1/testReport/
Max. process+thread count	2760 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

ayushtkn

Thanx @avijayanhwx for the work here, Looks Good Overall(The EditLog sync part assumed to be handled in a follow up). Couple of minor comments.

*Quota validations also need to be done, may be you can create a follow up JIRA for it, so as we don't forget it.

...ject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

...ject/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSwapBlockList.java

hadoop-yetus · 2020-01-31T02:30:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ HDFS-14978_ec_conversion Compile Tests _
+0 🆗	mvndep	1m 7s	Maven dependency ordering for branch
+1 💚	mvninstall	18m 8s	HDFS-14978_ec_conversion passed
+1 💚	compile	16m 46s	HDFS-14978_ec_conversion passed
+1 💚	checkstyle	2m 40s	HDFS-14978_ec_conversion passed
+1 💚	mvnsite	2m 47s	HDFS-14978_ec_conversion passed
+1 💚	shadedclient	19m 34s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 5s	HDFS-14978_ec_conversion passed
+0 🆗	spotbugs	2m 59s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	4m 58s	HDFS-14978_ec_conversion passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 52s	the patch passed
+1 💚	compile	15m 54s	the patch passed
+1 💚	javac	15m 54s	the patch passed
+1 💚	checkstyle	3m 30s	the patch passed
+1 💚	mvnsite	2m 51s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 26s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 7s	the patch passed
+1 💚	findbugs	5m 19s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 21s	hadoop-common in the patch passed.
-1 ❌	unit	93m 54s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 2s	The patch does not generate ASF License warnings.
		219m 48s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommissionWithBackoffMonitor

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/2/artifact/out/Dockerfile
GITHUB PR	#1819
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 92ad5cb4586f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	HDFS-14978_ec_conversion / `839e607`
Default Java	1.8.0_232
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/2/testReport/
Max. process+thread count	4136 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java

hadoop-yetus · 2020-01-31T23:13:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 46s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ HDFS-14978_ec_conversion Compile Tests _
+0 🆗	mvndep	1m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	23m 41s	HDFS-14978_ec_conversion passed
+1 💚	compile	21m 43s	HDFS-14978_ec_conversion passed
+1 💚	checkstyle	3m 25s	HDFS-14978_ec_conversion passed
+1 💚	mvnsite	3m 15s	HDFS-14978_ec_conversion passed
+1 💚	shadedclient	23m 42s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 12s	HDFS-14978_ec_conversion passed
+0 🆗	spotbugs	3m 33s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 55s	HDFS-14978_ec_conversion passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 11s	the patch passed
-1 ❌	compile	9m 32s	root in the patch failed.
-1 ❌	javac	9m 32s	root in the patch failed.
+1 💚	checkstyle	3m 14s	the patch passed
+1 💚	mvnsite	2m 47s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 38s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	3m 2s	the patch passed
-1 ❌	findbugs	0m 18s	hadoop-common in the patch failed.
-1 ❌	findbugs	0m 34s	hadoop-hdfs in the patch failed.
		_ Other Tests _
-1 ❌	unit	0m 27s	hadoop-common in the patch failed.
-1 ❌	unit	0m 17s	hadoop-hdfs in the patch failed.
+0 🆗	asflicense	0m 26s	ASF License check generated no output?
		123m 30s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/artifact/out/Dockerfile
GITHUB PR	#1819
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 24f7c44aa043 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	HDFS-14978_ec_conversion / `839e607`
Default Java	1.8.0_232
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/artifact/out/patch-compile-root.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/artifact/out/patch-compile-root.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/artifact/out/patch-findbugs-hadoop-common-project_hadoop-common.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/testReport/
Max. process+thread count	411 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-02-01T11:00:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	32m 57s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ HDFS-14978_ec_conversion Compile Tests _
+0 🆗	mvndep	1m 8s	Maven dependency ordering for branch
+1 💚	mvninstall	20m 56s	HDFS-14978_ec_conversion passed
+1 💚	compile	17m 41s	HDFS-14978_ec_conversion passed
+1 💚	checkstyle	2m 54s	HDFS-14978_ec_conversion passed
+1 💚	mvnsite	2m 37s	HDFS-14978_ec_conversion passed
+1 💚	shadedclient	20m 42s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 53s	HDFS-14978_ec_conversion passed
+0 🆗	spotbugs	3m 1s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 1s	HDFS-14978_ec_conversion passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 55s	the patch passed
+1 💚	compile	19m 16s	the patch passed
+1 💚	javac	19m 16s	the patch passed
+1 💚	checkstyle	2m 51s	the patch passed
+1 💚	mvnsite	2m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 59s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	2m 51s	the patch passed
+1 💚	findbugs	5m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 27s	hadoop-common in the patch passed.
-1 ❌	unit	113m 42s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 52s	The patch does not generate ASF License warnings.
		279m 6s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/4/artifact/out/Dockerfile
GITHUB PR	#1819
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c66b3e70ecd9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	HDFS-14978_ec_conversion / `839e607`
Default Java	1.8.0_242
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/4/testReport/
Max. process+thread count	3014 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1819/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

ayushtkn

LGTM +1

avijayanhwx · 2020-02-04T19:20:20Z

Thank you for the review @ayushtkn.

Aravindan Vijayan added 5 commits January 26, 2020 22:02

HDFS-14989. Add a 'swapBlockList' operation to Namenode.

c728a4d

HDFS-14989. Add a 'swapBlockList' operation to Namenode. (Fix checkst…

fb2fda8

…yle issues)

HDFS-14989. Address review comments.

ef331c8

HDFS-14989. Swap storage policy ID, check destination file genStamp.

2ca150e

HDFS-14989. Remove unused import.

eaee621

avijayanhwx requested review from arp7 and jojochuang January 27, 2020 06:07

ayushtkn requested changes Jan 27, 2020

View reviewed changes

HDFS-14989. Address review comment.

5a1e25d

ayushtkn reviewed Jan 31, 2020

View reviewed changes

...roject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SwapBlockListOp.java Outdated Show resolved Hide resolved

HDFS-14989. Use FSDirectory.resolveLastINode.

b5885b3

ayushtkn approved these changes Feb 1, 2020

View reviewed changes

avijayanhwx merged commit aad533d into apache:HDFS-14978_ec_conversion Feb 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-14989. Add a 'swapBlockList' operation to Namenode. #1819

HDFS-14989. Add a 'swapBlockList' operation to Namenode. #1819

Uh oh!

avijayanhwx commented Jan 27, 2020

Uh oh!

hadoop-yetus commented Jan 27, 2020

Uh oh!

ayushtkn left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jan 31, 2020

Uh oh!

Uh oh!

hadoop-yetus commented Jan 31, 2020

Uh oh!

hadoop-yetus commented Feb 1, 2020

Uh oh!

ayushtkn left a comment

Uh oh!

avijayanhwx commented Feb 4, 2020

Uh oh!

Uh oh!

HDFS-14989. Add a 'swapBlockList' operation to Namenode. #1819

HDFS-14989. Add a 'swapBlockList' operation to Namenode. #1819

Uh oh!

Conversation

avijayanhwx commented Jan 27, 2020

Uh oh!

hadoop-yetus commented Jan 27, 2020

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jan 31, 2020

Uh oh!

Uh oh!

hadoop-yetus commented Jan 31, 2020

Uh oh!

hadoop-yetus commented Feb 1, 2020

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

avijayanhwx commented Feb 4, 2020

Uh oh!

Uh oh!