HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed #4756

zhengchenyu · 2022-08-18T07:46:46Z

https://issues.apache.org/jira/browse/HDFS-16732

…k report is delayed.

hadoop-yetus · 2022-08-18T13:41:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	38m 5s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	1m 41s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 38s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 24s		trunk passed
+1 💚	mvnsite	1m 43s		trunk passed
+1 💚	javadoc	1m 23s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 46s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 48s		trunk passed
+1 💚	shadedclient	26m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 26s		the patch passed
+1 💚	compile	1m 28s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 28s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 0s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 111 unchanged - 0 fixed = 112 total (was 111)
+1 💚	mvnsite	1m 26s		the patch passed
+1 💚	javadoc	0m 59s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 32s		the patch passed
+1 💚	shadedclient	22m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	240m 24s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 13s		The patch does not generate ASF License warnings.
		353m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/1/artifact/out/Dockerfile
GITHUB PR	#4756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 43d62b41e7d7 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `187fe45`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/1/testReport/
Max. process+thread count	3343 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-08-19T10:30:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 31s		trunk passed
+1 💚	compile	1m 46s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 34s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 16s		trunk passed
+1 💚	mvnsite	1m 47s		trunk passed
+1 💚	javadoc	1m 13s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 39s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 34s		trunk passed
+1 💚	shadedclient	23m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 28s		the patch passed
+1 💚	compile	1m 29s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 29s		the patch passed
+1 💚	compile	1m 20s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	mvnsite	1m 27s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 26s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 21s		the patch passed
+1 💚	shadedclient	23m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	253m 23s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 14s		The patch does not generate ASF License warnings.
		364m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/2/artifact/out/Dockerfile
GITHUB PR	#4756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 4e3da7d52423 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c54f617`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/2/testReport/
Max. process+thread count	3480 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xkrogen

Agreed that the same logic from HDFS-12943 should be extended to getFileInfo/getListing. Thanks for putting up this PR!

For the test, can we just add new tests similar to TestObserverNode.testObserverNodeBlockMissingRetry? I am wondering if the new TestObserverNodeWhenReportDelay is overkill for this.

...s-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

hadoop-yetus · 2022-08-20T14:42:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 58s		trunk passed
+1 💚	compile	1m 36s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 27s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	1m 35s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 42s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 43s		trunk passed
+1 💚	shadedclient	25m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 37s		the patch passed
+1 💚	compile	1m 36s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 36s		the patch passed
+1 💚	compile	1m 25s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 8s		the patch passed
+1 💚	mvnsite	1m 36s		the patch passed
+1 💚	javadoc	1m 2s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 49s		the patch passed
+1 💚	shadedclient	26m 43s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	253m 2s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		370m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/3/artifact/out/Dockerfile
GITHUB PR	#4756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0ece423b518d 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `140b9e2`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/3/testReport/
Max. process+thread count	3005 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

...src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNodeWhenReportDelay.java

hadoop-yetus · 2022-08-25T11:07:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 27s		trunk passed
+1 💚	compile	1m 46s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 39s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 23s		trunk passed
+1 💚	mvnsite	1m 44s		trunk passed
+1 💚	javadoc	1m 22s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 45s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 8s		trunk passed
+1 💚	shadedclient	26m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 33s		the patch passed
+1 💚	compile	1m 38s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 38s		the patch passed
+1 💚	compile	1m 26s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 4s		the patch passed
+1 💚	mvnsite	1m 32s		the patch passed
+1 💚	javadoc	1m 3s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 36s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 38s		the patch passed
+1 💚	shadedclient	26m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	248m 54s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		369m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/4/artifact/out/Dockerfile
GITHUB PR	#4756
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux bcf6b87183ae 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bae43f8`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/4/testReport/
Max. process+thread count	3545 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xkrogen · 2022-08-25T17:45:27Z

New test is very clean :) Many thanks for the contribution @zhengchenyu !

I've merged this to trunk and branch-3.3.

…k report is delayed (#4756) Signed-off-by: Erik Krogen <xkrogen@apache.org> (cherry picked from commit 231a446)

…k report is delayed (apache#4756) Signed-off-by: Erik Krogen <xkrogen@apache.org>

…k report is delayed (apache#4756) Signed-off-by: Erik Krogen <xkrogen@apache.org> (cherry picked from commit 231a446)

HDFS-16732. [SBN READ] Avoid get location from observer when the bloc…

187fe45

…k report is delayed.

fix code style

c54f617

xkrogen reviewed Aug 19, 2022

View reviewed changes

...s-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java Outdated Show resolved Hide resolved

update code structure

140b9e2

xkrogen reviewed Aug 25, 2022

View reviewed changes

...src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNodeWhenReportDelay.java Outdated Show resolved Hide resolved

update the unit test

bae43f8

xkrogen changed the title ~~HDFS-16732. [SBN READ] Avoid get location from observer when the bloc…~~ HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed Aug 25, 2022

xkrogen merged commit 231a446 into apache:trunk Aug 25, 2022

xkrogen pushed a commit that referenced this pull request Aug 25, 2022

HDFS-16732. [SBN READ] Avoid get location from observer when the bloc…

3edddaf

…k report is delayed (#4756) Signed-off-by: Erik Krogen <xkrogen@apache.org> (cherry picked from commit 231a446)

zhengchenyu deleted the HDFS-16732 branch September 2, 2022 04:48

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16732. [SBN READ] Avoid get location from observer when the bloc…

da30c4c

…k report is delayed (apache#4756) Signed-off-by: Erik Krogen <xkrogen@apache.org>

LiuGuH pushed a commit to LiuGuH/hadoop that referenced this pull request Mar 29, 2024

HDFS-16732. [SBN READ] Avoid get location from observer when the bloc…

33b2ad6

…k report is delayed (apache#4756) Signed-off-by: Erik Krogen <xkrogen@apache.org> (cherry picked from commit 231a446)

LiuGuH pushed a commit to LiuGuH/hadoop that referenced this pull request Jul 4, 2024

HDFS-16732. [SBN READ] Avoid get location from observer when the bloc…

9b291ac

…k report is delayed (apache#4756) Signed-off-by: Erik Krogen <xkrogen@apache.org> (cherry picked from commit 231a446)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed #4756

HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed #4756

zhengchenyu commented Aug 18, 2022

hadoop-yetus commented Aug 18, 2022

hadoop-yetus commented Aug 19, 2022

xkrogen left a comment

hadoop-yetus commented Aug 20, 2022

hadoop-yetus commented Aug 25, 2022

xkrogen commented Aug 25, 2022

HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed #4756

HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed #4756

Conversation

zhengchenyu commented Aug 18, 2022

hadoop-yetus commented Aug 18, 2022

hadoop-yetus commented Aug 19, 2022

xkrogen left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Aug 20, 2022

hadoop-yetus commented Aug 25, 2022

xkrogen commented Aug 25, 2022