HADOOP-17272. ABFS Streams to support IOStatistics API #2604

mehakmeet · 2021-01-07T11:52:51Z

Previously #2353, Had to close it due to parent branch closing and pushed in a new PR. Now the IOStatistics has been merged in trunk as 3 PRs(#2577, #2579, #2580).

Tested using: mvn -T 1C -Dparallel-tests=abfs clean verify
Region: East US

[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0

[INFO] Results:
[ERROR] Tests run: 500, Failures: 0, Errors: 8, Skipped: 67

[errors related to #2331]

[INFO] Results:
[INFO]
[WARNING] Tests run: 255, Failures: 0, Errors: 0, Skipped: 77

steveloughran

LGTM; some minor comments, and checkstyle & javadocs need fixing.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsStatistic.java

...zure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStreamStatisticsImpl.java

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

...p-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatistics.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsInputStreamStatistics.java

mehakmeet · 2021-01-11T17:58:10Z

Checkstyle issues are due to static imports having ".*", but these are enums, should I go this route or add the class name before their name(adds a few bits in the code, but avoids checkstyle issue)?

Edit: Just saw some indentation issues as well, which my maven checkstyle didn't find. I'll update those in the new commit along with the change in static import if needed)

hadoop-yetus · 2021-01-12T10:08:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 0s		trunk passed
+1 💚	compile	23m 18s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	20m 57s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 54s		trunk passed
+1 💚	mvnsite	2m 26s		trunk passed
+1 💚	shadedclient	21m 1s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 48s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	2m 20s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 8s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 25s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	19m 32s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	19m 32s		the patch passed
+1 💚	compile	17m 16s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	17m 16s		the patch passed
+1 💚	checkstyle	2m 36s		the patch passed
+1 💚	mvnsite	2m 20s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 36s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 47s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	2m 20s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 41s		the patch passed
			_ Other Tests _
+1 💚	unit	9m 44s		hadoop-common in the patch passed.
+1 💚	unit	1m 47s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		195m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2604/3/artifact/out/Dockerfile
GITHUB PR	#2604
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0b3247abb854 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ca7dd5f`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2604/3/testReport/
Max. process+thread count	3217 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2604/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2021-01-12T11:38:50Z

...ools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java

 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;

+import org.apache.hadoop.fs.statistics.StreamStatisticNames;


nit: needs to go into the real org.apache block. The IDE is getting confused because of the third party stuff. as this is a "real" hadoop import it should go down below; when we backport the google ones will go back to their unshaded names

Sorry, my bad, should've double-checked.

steveloughran · 2021-01-12T11:39:25Z

...ure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java

@@ -18,32 +18,47 @@

 package org.apache.hadoop.fs.azurebfs.services;

+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.fs.statistics.StreamStatisticNames;


move into real apache group.

steveloughran

LGTM, just some import positioning. The move to shaded guava on trunk is complicating both imports and backporting, even though it will ultimately be good

steveloughran · 2021-01-12T11:40:48Z

+1 pending those imports

hadoop-yetus · 2021-01-12T14:57:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 58s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 57s		trunk passed
+1 💚	compile	20m 12s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	17m 26s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 40s		trunk passed
+1 💚	mvnsite	2m 24s		trunk passed
+1 💚	shadedclient	21m 2s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 48s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	2m 21s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 9s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 27s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	19m 25s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	19m 25s		the patch passed
+1 💚	compile	17m 36s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	17m 36s		the patch passed
+1 💚	checkstyle	2m 37s		the patch passed
+1 💚	mvnsite	2m 21s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 22s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 49s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	2m 19s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 41s		the patch passed
			_ Other Tests _
+1 💚	unit	9m 43s		hadoop-common in the patch passed.
+1 💚	unit	1m 47s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		187m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2604/4/artifact/out/Dockerfile
GITHUB PR	#2604
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 3d8bd27639b5 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ca7dd5f`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2604/4/testReport/
Max. process+thread count	3228 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2604/4/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Contributed by Mehakmeet Singh. Change-Id: I3445dec84b9b9e43bb1e41f709944ea05416bd74

HADOOP-17272. ABFS Streams to support IOStatistics API

f5e1205

mehakmeet mentioned this pull request Jan 7, 2021

HADOOP-17272. ABFS Streams to support IOStatistics API #2353

Closed

This comment has been minimized.

Sign in to view

steveloughran requested changes Jan 7, 2021

View reviewed changes

HADOOP-17272. review comments

542b842

This comment has been minimized.

Sign in to view

HADOOP-17272. checkstyle fix

b952932

steveloughran requested changes Jan 12, 2021

View reviewed changes

HADOOP-17272. import re-positioning

4355402

steveloughran merged commit 0a6ddfa into apache:trunk Jan 12, 2021

steveloughran mentioned this pull request Jan 12, 2021

Backport HADOOP-16830 to branch-3.3 #2616

Closed

asfgit pushed a commit that referenced this pull request Jan 22, 2021

HADOOP-17272. ABFS Streams to support IOStatistics API (#2604)

d20b2de

Contributed by Mehakmeet Singh. Change-Id: I3445dec84b9b9e43bb1e41f709944ea05416bd74

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17272. ABFS Streams to support IOStatistics API #2604

HADOOP-17272. ABFS Streams to support IOStatistics API #2604

Uh oh!

mehakmeet commented Jan 7, 2021

Uh oh!

This comment has been minimized.

steveloughran left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mehakmeet commented Jan 11, 2021 •

edited

Loading

Uh oh!

This comment has been minimized.

hadoop-yetus commented Jan 12, 2021

Uh oh!

steveloughran Jan 12, 2021

Uh oh!

mehakmeet Jan 12, 2021

Uh oh!

steveloughran Jan 12, 2021

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran commented Jan 12, 2021

Uh oh!

hadoop-yetus commented Jan 12, 2021

Uh oh!

Uh oh!

HADOOP-17272. ABFS Streams to support IOStatistics API #2604

HADOOP-17272. ABFS Streams to support IOStatistics API #2604

Uh oh!

Conversation

mehakmeet commented Jan 7, 2021

Uh oh!

This comment has been minimized.

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mehakmeet commented Jan 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

hadoop-yetus commented Jan 12, 2021

Uh oh!

steveloughran Jan 12, 2021

Choose a reason for hiding this comment

Uh oh!

mehakmeet Jan 12, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran Jan 12, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Jan 12, 2021

Uh oh!

hadoop-yetus commented Jan 12, 2021

Uh oh!

Uh oh!

mehakmeet commented Jan 11, 2021 •

edited

Loading