HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS #1990

mehakmeet · 2020-04-30T12:52:57Z

In some cases, ABFS-prefetch thread runs in the background which returns some bytes from the buffer and gives an extra readOp. Thus, making readOps values arbitrary and giving intermittent failures in some cases. Hence, readOps values of 2 or 3 are seen in different setups.

test runs:

mvn -T 1C -Dparallel-tests=abfs clean verify
mvn -T 1C -Dparallel-tests=abfs clean verify -Dtest=none -Dit.test=ITestAbfsStreamStatistics

Region: East US, West US

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 165.964 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (integration-test-abfs-parallel-classes) @ hadoop-azure ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 256.797 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (integration-test-abfs-parallel-classesAndMethods) @ hadoop-azure ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (integration-test-abfs-parallel-classes) @ hadoop-azure ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:14 min (Wall Clock)
[INFO] Finished at: 2020-04-30T18:01:26+05:30

hadoop-yetus · 2020-04-30T14:37:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 53s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	31m 3s	trunk passed
+1 💚	compile	0m 45s	trunk passed
+1 💚	checkstyle	0m 29s	trunk passed
+1 💚	mvnsite	0m 46s	trunk passed
+1 💚	shadedclient	20m 20s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s	trunk passed
+0 🆗	spotbugs	1m 19s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 15s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s	the patch passed
+1 💚	compile	0m 29s	the patch passed
+1 💚	javac	0m 29s	the patch passed
+1 💚	checkstyle	0m 22s	the patch passed
+1 💚	mvnsite	0m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	19m 26s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 35s	the patch passed
+1 💚	findbugs	1m 54s	the patch passed
		_ Other Tests _
-1 ❌	unit	1m 51s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 59s	The patch does not generate ASF License warnings.
		85m 54s

Reason	Tests
Failed junit tests	hadoop.fs.azure.TestClientThrottlingAnalyzer

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/artifact/out/Dockerfile
GITHUB PR	#1990
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 51d371e18d22 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b5b45c5`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/testReport/
Max. process+thread count	366 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStreamStatistics.java

steveloughran · 2020-05-07T11:14:17Z

+1, LGTM. IF it fails now, we'd know where to begin debugging it.

hadoop-yetus · 2020-05-07T11:42:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 14s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 15s	trunk passed
+1 💚	compile	0m 27s	trunk passed
+1 💚	checkstyle	0m 21s	trunk passed
+1 💚	mvnsite	0m 32s	trunk passed
+1 💚	shadedclient	16m 31s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	trunk passed
+0 🆗	spotbugs	0m 56s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 54s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	mvnsite	0m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 32s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 21s	the patch passed
+1 💚	findbugs	1m 10s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 29s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 51s	The patch does not generate ASF License warnings.
		64m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/2/artifact/out/Dockerfile
GITHUB PR	#1990
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f961982a26d6 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `99840aa`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/2/testReport/
Max. process+thread count	343 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1990/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…FS (#1990) Contributed by: Mehakmeet Singh In some cases, ABFS-prefetch thread runs in the background which returns some bytes from the buffer and gives an extra readOp. Thus, making readOps values arbitrary and giving intermittent failures in some cases. Hence, readOps values of 2 or 3 are seen in different setups.

…istics in ABFS (apache#1990) Contributed by: Mehakmeet Singh In some cases, ABFS-prefetch thread runs in the background which returns some bytes from the buffer and gives an extra readOp. Thus, making readOps values arbitrary and giving intermittent failures in some cases. Hence, readOps values of 2 or 3 are seen in different setups. Change-Id: Id6c0bc989dcc506a035f081a7d6e89f8d64a1037

HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS

e2e29be

mehakmeet mentioned this pull request Apr 30, 2020

HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO. #1918

Merged

steveloughran requested changes May 6, 2020

View reviewed changes

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStreamStatistics.java Outdated Show resolved Hide resolved

HADOOP-17018. Actual value in error message

19a314a

mehakmeet requested a review from steveloughran May 7, 2020 11:01

steveloughran merged commit 192cad9 into apache:trunk May 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS #1990

HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS #1990

Uh oh!

mehakmeet commented Apr 30, 2020

Uh oh!

hadoop-yetus commented Apr 30, 2020

Uh oh!

Uh oh!

steveloughran commented May 7, 2020

Uh oh!

hadoop-yetus commented May 7, 2020

Uh oh!

Uh oh!

HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS #1990

HADOOP-17018. Intermittent failing of ITestAbfsStreamStatistics in ABFS #1990

Uh oh!

Conversation

mehakmeet commented Apr 30, 2020

Uh oh!

hadoop-yetus commented Apr 30, 2020

Uh oh!

Uh oh!

steveloughran commented May 7, 2020

Uh oh!

hadoop-yetus commented May 7, 2020

Uh oh!

Uh oh!