HDFS-16318. Add exception blockinfo #3649

GuoPhilipse · 2021-11-12T08:42:47Z

we may suffer Could not obtain the last block location exception, but we may reading more than one file, the following exception cannnot guide us to find the problem block or dn info.

2021-11-12 14:01:59,633 WARN [main] org.apache.hadoop.hdfs.DFSClient: Last block locations not available. Datanodes might not have reported blocks completely. Will retry for 3 times
2021-11-12 14:02:03,724 WARN [main] org.apache.hadoop.hdfs.DFSClient: Last block locations not available. Datanodes might not have reported blocks completely. Will retry for 2 times
2021-11-12 14:02:07,726 WARN [main] org.apache.hadoop.hdfs.DFSClient: Last block locations not available. Datanodes might not have reported blocks completely. Will retry for 1 times

Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:251) ... 11 more
Caused by: java.io.IOException: Could not obtain the last block locations. at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:291) at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:264) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1535) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:162) at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:261) at org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:463) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:768) at org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:109) at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:66) ... 15 more

sync

hadoop-yetus · 2021-11-12T10:25:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 4s		trunk passed
+1 💚	compile	1m 0s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 49s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 25s		trunk passed
+1 💚	mvnsite	0m 57s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 36s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 33s		trunk passed
+1 💚	shadedclient	24m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 52s		the patch passed
+1 💚	compile	0m 53s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 53s		the patch passed
+1 💚	compile	0m 43s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 18s		the patch passed
+1 💚	mvnsite	0m 46s		the patch passed
+1 💚	javadoc	0m 31s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 29s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 36s		the patch passed
+1 💚	shadedclient	24m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 16s		hadoop-hdfs-client in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		101m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/1/artifact/out/Dockerfile
GITHUB PR	#3649
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 41fb566a625b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c389fa9`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/1/testReport/
Max. process+thread count	606 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java

hadoop-yetus · 2021-11-14T15:14:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 8s		trunk passed
+1 💚	compile	1m 0s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 26s		trunk passed
+1 💚	mvnsite	0m 56s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 36s		trunk passed
+1 💚	shadedclient	24m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
+1 💚	compile	0m 52s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 52s		the patch passed
+1 💚	compile	0m 43s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 18s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 8 new + 31 unchanged - 0 fixed = 39 total (was 31)
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 29s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	spotbugs	2m 40s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.html	hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	24m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 15s		hadoop-hdfs-client in the patch passed.
-1 ❌	asflicense	0m 30s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		101m 7s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs-client
	Inconsistent synchronization of org.apache.hadoop.hdfs.DFSInputStream.currentNode; locked 94% of time Unsynchronized access at DFSInputStream.java:94% of time Unsynchronized access at DFSInputStream.java:[line 260]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/2/artifact/out/Dockerfile
GITHUB PR	#3649
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux c160878232c5 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `402c386`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/2/testReport/
Max. process+thread count	615 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

…blockinfo

hadoop-yetus · 2021-11-18T13:40:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 1s		trunk passed
+1 💚	compile	0m 59s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 52s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 26s		trunk passed
+1 💚	mvnsite	0m 57s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 34s		trunk passed
+1 💚	shadedclient	24m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s		the patch passed
+1 💚	compile	0m 54s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 54s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 17s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt	hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 8 new + 31 unchanged - 0 fixed = 39 total (was 31)
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 29s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	spotbugs	2m 41s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.html	hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	24m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 16s		hadoop-hdfs-client in the patch passed.
-1 ❌	asflicense	0m 29s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		101m 5s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs-client
	Inconsistent synchronization of org.apache.hadoop.hdfs.DFSInputStream.currentNode; locked 94% of time Unsynchronized access at DFSInputStream.java:94% of time Unsynchronized access at DFSInputStream.java:[line 260]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/3/artifact/out/Dockerfile
GITHUB PR	#3649
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 428d93d2f5cd 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7e8a3de`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/3/testReport/
Max. process+thread count	600 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3649/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn

There are some complains from Jenkins, That needs to be checked. Regarding the test for logs. Can use :
GenericTestUtils.LogCapturer.captureLogs(
Can find usage in the code and do similarly.
In general the log change makes sense to me.

GuoPhilipse · 2021-12-04T17:30:31Z

There are some complains from Jenkins, That needs to be checked. Regarding the test for logs. Can use : GenericTestUtils.LogCapturer.captureLogs( Can find usage in the code and do similarly. In general the log change makes sense to me.

Thanks @ayushtkn for your comment, the complains seems not related with the changing code, it seems a little trouble to test this PR, emm , anyway i will try to fix it.

ayushtkn · 2021-12-07T08:53:58Z

hmm, not sure why is it happening, try a rebase on top of present trunk, might help, just a wild guess...

GuoPhilipse · 2021-12-07T10:16:39Z

hmm, not sure why is it happening, try a rebase on top of present trunk, might help, just a wild guess...

Thanks @ayushtkn for you advice ,will try it later.

GuoPhilipse and others added 8 commits November 29, 2020 05:02

Merge pull request #1 from apache/trunk

3c51579

sync

Merge pull request #2 from apache/trunk

a449d04

sync

Merge pull request #3 from apache/trunk

d4f968b

sync

Merge pull request #4 from apache/trunk

d522abf

sync

Merge pull request #5 from apache/trunk

de633b8

sync

Merge pull request #6 from apache/trunk

66ce06b

sync

'addblockinfo'

359bee5

'addblockinfo'

c389fa9

Hexiaoqiao reviewed Nov 14, 2021

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java Outdated Show resolved Hide resolved

GuoPhilipse added 2 commits November 14, 2021 21:30

fixcomment

d956ed4

fixcomment

402c386

tclxgf13871 added 2 commits November 18, 2021 19:53

'triggerrebuild'

305f6b4

Merge branch 'addblockinfo' of github.com:GuoPhilipse/hadoop into add…

7e8a3de

…blockinfo

ayushtkn reviewed Dec 4, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16318. Add exception blockinfo #3649

HDFS-16318. Add exception blockinfo #3649

GuoPhilipse commented Nov 12, 2021

Uh oh!

hadoop-yetus commented Nov 12, 2021

Uh oh!

Uh oh!

hadoop-yetus commented Nov 14, 2021

Uh oh!

hadoop-yetus commented Nov 18, 2021

Uh oh!

ayushtkn left a comment

Uh oh!

GuoPhilipse commented Dec 4, 2021 •

edited

Loading

Uh oh!

ayushtkn commented Dec 7, 2021

Uh oh!

GuoPhilipse commented Dec 7, 2021

Uh oh!

Uh oh!

HDFS-16318. Add exception blockinfo #3649

Are you sure you want to change the base?

HDFS-16318. Add exception blockinfo #3649

Conversation

GuoPhilipse commented Nov 12, 2021

Uh oh!

hadoop-yetus commented Nov 12, 2021

Uh oh!

Uh oh!

hadoop-yetus commented Nov 14, 2021

Uh oh!

hadoop-yetus commented Nov 18, 2021

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

GuoPhilipse commented Dec 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayushtkn commented Dec 7, 2021

Uh oh!

GuoPhilipse commented Dec 7, 2021

Uh oh!

Uh oh!

GuoPhilipse commented Dec 4, 2021 •

edited

Loading