HADOOP-17113. Adding ReadAhead Counters in ABFS #2154

mehakmeet · 2020-07-20T15:46:41Z

tested by: mvn -T 1C -Dparallel-tests=abfs clean verify
Region: East US, West US

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16:11 min (Wall Clock)
[INFO] Finished at: 2020-07-20T20:39:21+05:30

hadoop-yetus · 2020-07-20T17:35:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	30m 11s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	26m 23s	trunk passed
+1 💚	compile	0m 40s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	0m 32s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 21s	trunk passed
+1 💚	mvnsite	0m 34s	trunk passed
+1 💚	shadedclient	16m 23s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 24s	hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 23s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	0m 54s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 53s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 27s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	0m 27s	the patch passed
+1 💚	compile	0m 25s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	0m 25s	the patch passed
+1 💚	checkstyle	0m 15s	the patch passed
+1 💚	mvnsite	0m 29s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 21s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 21s	hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 20s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	0m 54s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 13s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 27s	The patch does not generate ASF License warnings.
		99m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/1/artifact/out/Dockerfile
GITHUB PR	#2154
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 5ff1c5366f44 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6cbd885`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/1/testReport/
Max. process+thread count	315 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2020-07-20T19:54:52Z

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsInputStreamStatistics.java

+      }
+
+      /*
+       * Verifying the counter values of readAheadBytesRead and remoteBytesRead.


nice explanation

steveloughran

Patch LGTM

I've suggested using AssertJ assertions, as they are a lot better for things like > asserts, but that is entirely optional -the patch is ready to go in even without it

So: your call. Do you want to play with AssertJ or do you just want to ship it as is and play with AssertJ on your next bit of work?

s

steveloughran · 2020-07-20T19:55:36Z

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsInputStreamStatistics.java

+       * remotely could also be greater than 32Kb.
+       *
+       */
+      assertTrue(String.format("actual value of %d is not greater than or "


Try using AssertJ.assertThat here, it lets you declare the specific "isGreaterThan" assertion; it's describedAs() does the string formatting too.

It would be good to add it in this patch. Thanks for the tip.

hadoop-yetus · 2020-07-21T06:15:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 1s	trunk passed
+1 💚	compile	0m 39s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	0m 34s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 36s	trunk passed
+1 💚	shadedclient	14m 58s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 30s	hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 27s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	0m 55s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 52s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s	the patch passed
+1 💚	compile	0m 29s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	0m 29s	the patch passed
+1 💚	compile	0m 25s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	0m 25s	the patch passed
+1 💚	checkstyle	0m 17s	the patch passed
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 48s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 26s	hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 25s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	0m 57s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 24s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		60m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/2/artifact/out/Dockerfile
GITHUB PR	#2154
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2211ea753785 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3833c61`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/2/testReport/
Max. process+thread count	413 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2020-07-21T10:19:03Z

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsInputStreamStatistics.java

+          .isGreaterThanOrEqualTo(CUSTOM_READ_AHEAD_BUFFER_SIZE);
+
+    } catch (InterruptedException e) {
+      e.printStackTrace();


can't we just throw this? If not, at least use LOG

steveloughran · 2020-07-21T10:20:17Z

ok, looks good. Just remove that catch of interrupted exceptions and instead add that exception to the list of exceptions the test can throw

hadoop-yetus · 2020-07-21T13:35:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 47s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	26m 16s	trunk passed
+1 💚	compile	0m 46s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	0m 37s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 26s	trunk passed
+1 💚	mvnsite	0m 38s	trunk passed
+1 💚	shadedclient	17m 52s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 27s	hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 25s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	1m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 3s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s	the patch passed
+1 💚	compile	0m 28s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	0m 28s	the patch passed
+1 💚	compile	0m 28s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	0m 28s	the patch passed
+1 💚	checkstyle	0m 18s	the patch passed
+1 💚	mvnsite	0m 30s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	16m 40s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 27s	hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 23s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	1m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 33s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 32s	The patch does not generate ASF License warnings.
		75m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/3/artifact/out/Dockerfile
GITHUB PR	#2154
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f0efb2c8bebd 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `d57462f`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/3/testReport/
Max. process+thread count	293 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2154/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2020-07-22T17:22:53Z

+1, merged to trunk

steveloughran · 2020-07-22T17:24:20Z

..doesn't merge to 3.3. Have there been some trunk changes I've missed?

mehakmeet · 2020-07-22T17:58:52Z

This one #2147(Backport I have already opened) and #2076 and #2056 (merged in trunk but needs backporting).

Contributed by Mehakmeet Singh

Contributed by Mehakmeet Singh Change-Id: I6bbd8165385a9267ed64831bb1efa18b6554feb1

Contributed by Mehakmeet Singh

Contributed by Mehakmeet Singh Change-Id: Iaa85ce9c48c771f5a5153c5419190088a19e8b5e

HADOOP-17113. Adding ReadAhead Counters in ABFS

ba2869a

steveloughran reviewed Jul 20, 2020

View reviewed changes

HADOOP-17113. Adding assertThat

2f3763d

mehakmeet requested a review from steveloughran July 21, 2020 05:48

steveloughran reviewed Jul 21, 2020

View reviewed changes

HADOOP-17113. adding exception to method

84b282c

mehakmeet requested a review from steveloughran July 21, 2020 11:23

steveloughran merged commit 48a7c5b into apache:trunk Jul 22, 2020

mehakmeet added a commit to mehakmeet/hadoop that referenced this pull request Sep 8, 2020

HADOOP-17113. Adding ReadAhead Counters in ABFS (apache#2154)

d6008ad

Contributed by Mehakmeet Singh

mehakmeet mentioned this pull request Sep 8, 2020

HADOOP-17113, HADOOP-17158. Adding ReadAhead Counters and fixing ITestAbfsInputStreamStatistics#testReadAheadCounters #2282

Closed

asfgit pushed a commit that referenced this pull request Sep 8, 2020

HADOOP-17113. Adding ReadAhead Counters in ABFS (#2154)

28f1ded

Contributed by Mehakmeet Singh Change-Id: I6bbd8165385a9267ed64831bb1efa18b6554feb1

snvijaya pushed a commit to snvijaya/hadoop that referenced this pull request May 23, 2021

HADOOP-17113. Adding ReadAhead Counters in ABFS (apache#2154)

73bc041

Contributed by Mehakmeet Singh

jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023

CDPD-8525, HADOOP-17113. Adding ReadAhead Counters in ABFS (apache#2154)

7dd9e93

Contributed by Mehakmeet Singh Change-Id: Iaa85ce9c48c771f5a5153c5419190088a19e8b5e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17113. Adding ReadAhead Counters in ABFS #2154

HADOOP-17113. Adding ReadAhead Counters in ABFS #2154

Uh oh!

mehakmeet commented Jul 20, 2020 •

edited

Loading

Uh oh!

hadoop-yetus commented Jul 20, 2020

Uh oh!

steveloughran Jul 20, 2020

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran Jul 20, 2020

Uh oh!

mehakmeet Jul 21, 2020

Uh oh!

hadoop-yetus commented Jul 21, 2020

Uh oh!

steveloughran Jul 21, 2020

Uh oh!

steveloughran commented Jul 21, 2020

Uh oh!

hadoop-yetus commented Jul 21, 2020

Uh oh!

steveloughran commented Jul 22, 2020

Uh oh!

steveloughran commented Jul 22, 2020

Uh oh!

mehakmeet commented Jul 22, 2020

Uh oh!

Uh oh!

HADOOP-17113. Adding ReadAhead Counters in ABFS #2154

HADOOP-17113. Adding ReadAhead Counters in ABFS #2154

Uh oh!

Conversation

mehakmeet commented Jul 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Jul 20, 2020

Uh oh!

steveloughran Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

mehakmeet Jul 21, 2020

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jul 21, 2020

Uh oh!

steveloughran Jul 21, 2020

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Jul 21, 2020

Uh oh!

hadoop-yetus commented Jul 21, 2020

Uh oh!

steveloughran commented Jul 22, 2020

Uh oh!

steveloughran commented Jul 22, 2020

Uh oh!

mehakmeet commented Jul 22, 2020

Uh oh!

Uh oh!

mehakmeet commented Jul 20, 2020 •

edited

Loading