HADOOP-17038 Support positional read in AbfsInputStream

[anoopsjohn]

No description provided.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	34m 3s	trunk passed
+1 💚	compile	0m 34s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	0m 35s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	0m 24s	trunk passed
+1 💚	mvnsite	0m 38s	trunk passed
+1 💚	shadedclient	18m 0s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 30s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	0m 29s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	1m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 3s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s	the patch passed
+1 💚	compile	0m 33s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	0m 33s	the patch passed
+1 💚	compile	0m 30s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	0m 30s	the patch passed
-0 ⚠️	checkstyle	0m 20s	hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
+1 💚	mvnsite	0m 33s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	16m 35s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 26s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	0m 24s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	findbugs	1m 4s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 36s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 36s	The patch does not generate ASF License warnings.
		82m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/artifact/out/Dockerfile
GITHUB PR	#2206
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e1b6b117f190 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 5e0f879
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/testReport/
Max. process+thread count	420 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[anoopsjohn]

ABFS tests run on East US region storage. (Excluded ITestAzureBlobFileSystemAuthorization as I had issues running that. I dont think this feature will have any impact by this patch)

mvn -T 1C -Dparallel-tests=abfs clean verify
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:28 min (Wall Clock)
[INFO] Finished at: 2020-08-10T05:57:51+00:00
[INFO] Final Memory: 63M/1254M

[anoopsjohn]

Ping @steveloughran , @snvijaya.. Pls help to review.

[steveloughran]

As discussed, I don't like this design.

It's optimised purely for the current API uses which HBase makes, is brittle to any changes there and doesn't provide any way to explicitly tune things.

Please add openFile to the ABFS Store and use that as a way to explicitly allow the caller to specify positioned read policy; HBase can then use the right option at the right point. We could backport that API to hadoop-3.2.x if that would aid there

[anoopsjohn]

Sorry for the delay in responding in the jira. Thanks for the pointer. As said in jira, if per FIS it can be controlled, that is best choice. I was not knowing such thing added.. Was looking at 3.1.x code line.
So your suggestion is to use openFile(Path path) API and on the FutureDataInputStreamBuilder use opt(String , boolean) or such APIs. From HBase we know which FIS will always use pread. Sounds like a plan. The Keys for the FSBuilder APIs needs to be generic. Any pointer where we need to add this? May be already we have a place for this with sample keys .. Wanted to see the naming convention. I can refer that. Thanks again Steve.

[steveloughran]

The Keys for the FSBuilder APIs needs to be generic. Any pointer where we need to add this? May be already we have a place for this with sample keys .. Wanted to see the naming convention. I can refer that. Thanks again Steve.

Have a look at what I'm doing in #2168 for this.

[steveloughran]

why the close? I know it didn't quite work as is, but we should see what could be lifted/merged, in particular: tests

[anoopsjohn]

oh sorry.. i closed my patch branch.. i will give new version based on openFile() way as discussed above.. Will include all tests in that too. Started working on new patch