HADOOP-16965 Refactor abfs stream configuration. #1956

mukund-thakur · 2020-04-14T09:48:38Z

Ran all abfs tests using mvn -T 1C -Dparallel-tests=abfs clean verify with my storage account in us-east.

mukund-thakur · 2020-04-14T09:48:55Z

@steveloughran Does this makes sense?

hadoop-yetus · 2020-04-14T11:20:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 3s	trunk passed
+1 💚	compile	0m 28s	trunk passed
+1 💚	checkstyle	0m 20s	trunk passed
+1 💚	mvnsite	0m 31s	trunk passed
+1 💚	shadedclient	16m 28s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	trunk passed
+0 🆗	spotbugs	1m 1s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 58s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	mvnsite	0m 26s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 51s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 25s	the patch passed
+1 💚	findbugs	1m 9s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 10s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 29s	The patch does not generate ASF License warnings.
		64m 43s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/1/artifact/out/Dockerfile
GITHUB PR	#1956
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e831e2886726 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `aeeebc5`
Default Java	1.8.0_242
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/1/testReport/
Max. process+thread count	312 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-04-15T15:03:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	26m 10s	trunk passed
+1 💚	compile	0m 33s	trunk passed
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 36s	trunk passed
+1 💚	shadedclient	19m 7s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 25s	trunk passed
+0 🆗	spotbugs	1m 0s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 57s	trunk passed
-0 ⚠️	patch	1m 20s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s	the patch passed
+1 💚	compile	0m 29s	the patch passed
+1 💚	javac	0m 29s	the patch passed
-0 ⚠️	checkstyle	0m 19s	hadoop-tools/hadoop-azure: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	18m 45s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	the patch passed
+1 💚	findbugs	1m 9s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 35s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 38s	The patch does not generate ASF License warnings.
		77m 4s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/2/artifact/out/Dockerfile
GITHUB PR	#1956
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0cb8fdcf24e5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `4db598e`
Default Java	1.8.0_242
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/2/testReport/
Max. process+thread count	309 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM, a few minor nits but otherwise I'm happy.

+1 pending those changes -if you can do them quickly I'll merge as soon as I can. Thanks

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java

steveloughran · 2020-04-20T15:35:43Z

...doop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamContext.java

+
+  public AbfsOutputStreamContext build() {
+    // Validation of parameters to be done here.
+    return this;


do we need any validators here yet, or is there enough in the code as it is?

Now we don't have any validation yet but we can add the new ones here in future if required.

...adoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStreamContext.java

hadoop-yetus · 2020-04-21T14:48:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	20m 59s	trunk passed
+1 💚	compile	0m 30s	trunk passed
+1 💚	checkstyle	0m 22s	trunk passed
+1 💚	mvnsite	0m 34s	trunk passed
+1 💚	shadedclient	15m 15s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	trunk passed
+0 🆗	spotbugs	0m 54s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 52s	trunk passed
-0 ⚠️	patch	1m 12s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
+1 💚	checkstyle	0m 17s	the patch passed
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 25s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	the patch passed
+1 💚	findbugs	0m 54s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 23s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 32s	The patch does not generate ASF License warnings.
		61m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/3/artifact/out/Dockerfile
GITHUB PR	#1956
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 00a114c67f04 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `60fa153`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/3/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1956/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Contributed by Mukund Thakur.

Contributed by Mukund Thakur. (cherry picked from commit 8031c66)

Contributed by Mukund Thakur.

…#1956) Contributed by Mukund Thakur. Conflicts: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java Change-Id: I41c454dbc1d6be7c88a58c9edd88aadf3766caa5

HADOOP-16965 Rafactor abfs stream configuration.

3ed1ed1

Changing to builder based contexts

492ca11

steveloughran requested changes Apr 20, 2020

View reviewed changes

This was referenced Apr 20, 2020

HADOOP-16961. ABFS: Adding metrics to AbfsInputStream #1946

Closed

HADOOP-16916: ABFS: Delegation SAS generator for integration with Ranger #1965

Closed

Review comments

8faf18c

mukund-thakur requested a review from steveloughran April 21, 2020 13:31

steveloughran merged commit 8031c66 into apache:trunk Apr 21, 2020

asfgit pushed a commit that referenced this pull request Apr 22, 2020

HADOOP-16965. Refactor abfs stream configuration. (#1956)

98fdbb8

Contributed by Mukund Thakur.

snvijaya pushed a commit to snvijaya/hadoop that referenced this pull request May 23, 2021

HADOOP-16965. Refactor abfs stream configuration. (apache#1956)

8c685dc

Contributed by Mukund Thakur.

arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 11, 2022

HADOOP-16965. Refactor abfs stream configuration. (apache#1956)

71f5619

Contributed by Mukund Thakur. (cherry picked from commit 8031c66)

arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 14, 2022

HADOOP-16965. Refactor abfs stream configuration. (apache#1956)

dba3321

Contributed by Mukund Thakur. (cherry picked from commit 8031c66)

arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request May 3, 2022

HADOOP-16965. Refactor abfs stream configuration. (apache#1956)

b03b211

Contributed by Mukund Thakur. (cherry picked from commit 8031c66)

zhangxiping1 pushed a commit to zhangxiping1/hadoop that referenced this pull request Dec 13, 2022

HADOOP-16965. Refactor abfs stream configuration. (apache#1956)

eac975b

Contributed by Mukund Thakur.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16965 Refactor abfs stream configuration. #1956

HADOOP-16965 Refactor abfs stream configuration. #1956

Uh oh!

mukund-thakur commented Apr 14, 2020

Uh oh!

mukund-thakur commented Apr 14, 2020

Uh oh!

hadoop-yetus commented Apr 14, 2020

Uh oh!

hadoop-yetus commented Apr 15, 2020

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

steveloughran Apr 20, 2020

Uh oh!

mukund-thakur Apr 21, 2020

Uh oh!

Uh oh!

hadoop-yetus commented Apr 21, 2020

Uh oh!

Uh oh!

HADOOP-16965 Refactor abfs stream configuration. #1956

HADOOP-16965 Refactor abfs stream configuration. #1956

Uh oh!

Conversation

mukund-thakur commented Apr 14, 2020

Uh oh!

mukund-thakur commented Apr 14, 2020

Uh oh!

hadoop-yetus commented Apr 14, 2020

Uh oh!

hadoop-yetus commented Apr 15, 2020

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

mukund-thakur Apr 21, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hadoop-yetus commented Apr 21, 2020

Uh oh!

Uh oh!