HDFS-16552. Fix NPE for TestBlockManager #4210

tomscut · 2022-04-21T10:04:51Z

Fix NPE for BlockManager#scheduleReconstruction.

There is a NPE in BlockManager when run TestBlockManager#testSkipReconstructionWithManyBusyNodes2. Because NameNodeMetrics is not initialized in this unit test.

Related ci link, see this.

hadoop-yetus · 2022-04-21T18:50:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 43s		trunk passed
+1 💚	compile	1m 41s		trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 30s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 17s		trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 47s		trunk passed
+1 💚	shadedclient	26m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 21s		the patch passed
+1 💚	compile	1m 27s		the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 27s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 0s		the patch passed
+1 💚	mvnsite	1m 27s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 34s		the patch passed
+1 💚	shadedclient	26m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	405m 3s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		524m 36s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand
	hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/1/artifact/out/Dockerfile
GITHUB PR	#4210
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 90793ca6794e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / fd09c81a270b07aadca0795c0bdb80c167ff6313
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/1/testReport/
Max. process+thread count	2055 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-04-22T01:37:54Z

Hi @tasanuma , could you please take a look at this. Thank you.

tasanuma · 2022-04-22T03:18:59Z

@tomscut Thanks for reporting it. I'm not sure why CI didn't catch it in #4138.

Instead of fixing BlockManager, I prefer to fix the unit test. How about adding the following line to the top of testSkipReconstructionWithManyBusyNodes() and testSkipReconstructionWithManyBusyNodes2()?

    NameNode.initMetrics(new Configuration(), HdfsServerConstants.NamenodeRole.NAMENODE);

tomscut · 2022-04-22T04:36:35Z

NameNode.initMetrics(new Configuration(), HdfsServerConstants.NamenodeRole.NAMENODE);

Thanks @tasanuma for your suggestions. I will update it.

tasanuma · 2022-04-22T05:12:55Z

+1, pending Jenkins.

hadoop-yetus · 2022-04-22T13:07:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 51s		trunk passed
+1 💚	compile	1m 39s		trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 31s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 18s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 20s		trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 44s		trunk passed
+1 💚	shadedclient	26m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	1m 27s		the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 27s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 2s		the patch passed
+1 💚	mvnsite	1m 24s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 32s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 29s		the patch passed
+1 💚	shadedclient	25m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	359m 6s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		477m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/2/artifact/out/Dockerfile
GITHUB PR	#4210
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux e4f25f7f85a6 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c721b5c`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/2/testReport/
Max. process+thread count	1998 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tasanuma · 2022-04-23T03:14:25Z

Thanks for fixing it, @tomscut!

(cherry picked from commit 5ebbacc)

tomscut · 2022-04-23T03:20:37Z

Thank you @tasanuma .

tomscut changed the title ~~HDFS-16552. Fix NPE for BlockManager#scheduleReconstruction~~ HDFS-16552. Fix NPE for TestBlockManager Apr 22, 2022

HDFS-16552. Fix NPE for TestBlockManager

c721b5c

tomscut force-pushed the HDFS-16552 branch from fd09c81 to c721b5c Compare April 22, 2022 05:08

tasanuma approved these changes Apr 23, 2022

View reviewed changes

tasanuma merged commit 5ebbacc into apache:trunk Apr 23, 2022

tasanuma pushed a commit that referenced this pull request Apr 23, 2022

HDFS-16552. Fix NPE for TestBlockManager (#4210)

9132eeb

(cherry picked from commit 5ebbacc)

tasanuma pushed a commit that referenced this pull request Apr 23, 2022

HDFS-16552. Fix NPE for TestBlockManager (#4210)

0965822

(cherry picked from commit 5ebbacc)

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16552. Fix NPE for TestBlockManager (apache#4210)

a6de365

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16552. Fix NPE for TestBlockManager #4210

HDFS-16552. Fix NPE for TestBlockManager #4210

tomscut commented Apr 21, 2022

hadoop-yetus commented Apr 21, 2022

tomscut commented Apr 22, 2022

tasanuma commented Apr 22, 2022

tomscut commented Apr 22, 2022

tasanuma commented Apr 22, 2022

hadoop-yetus commented Apr 22, 2022

tasanuma commented Apr 23, 2022

tomscut commented Apr 23, 2022

HDFS-16552. Fix NPE for TestBlockManager #4210

HDFS-16552. Fix NPE for TestBlockManager #4210

Conversation

tomscut commented Apr 21, 2022

hadoop-yetus commented Apr 21, 2022

tomscut commented Apr 22, 2022

tasanuma commented Apr 22, 2022

tomscut commented Apr 22, 2022

tasanuma commented Apr 22, 2022

hadoop-yetus commented Apr 22, 2022

tasanuma commented Apr 23, 2022

tomscut commented Apr 23, 2022