HDDS-1780. TestFailureHandlingByClient tests are flaky. #1073

bshashikant · 2019-07-10T13:12:37Z

No description provided.

supratimdeka

+1
LGTM

hadoop-yetus · 2019-07-16T12:51:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	11	#1073 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#1073
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bshashikant · 2019-07-16T12:57:31Z

/retest

hadoop-yetus · 2019-07-16T15:37:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	134	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	75	Maven dependency ordering for branch
+1	mvninstall	648	trunk passed
+1	compile	308	trunk passed
+1	checkstyle	78	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	951	branch has no errors when building and testing our client artifacts.
+1	javadoc	167	trunk passed
0	spotbugs	335	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	546	trunk passed
-0	patch	377	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
0	mvndep	31	Maven dependency ordering for patch
+1	mvninstall	455	the patch passed
+1	compile	266	the patch passed
+1	javac	266	the patch passed
+1	checkstyle	75	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	745	patch has no errors when building and testing our client artifacts.
+1	javadoc	174	the patch passed
+1	findbugs	547	the patch passed
		_ Other Tests _
+1	unit	343	hadoop-hdds in the patch passed.
-1	unit	2421	hadoop-ozone in the patch failed.
+1	asflicense	43	The patch does not generate ASF License warnings.
		8182

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.hdds.scm.pipeline.TestNodeFailure
	hadoop.ozone.client.rpc.TestFailureHandlingByClient
	hadoop.hdds.scm.pipeline.TestSCMRestart
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
	hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
	hadoop.hdds.scm.pipeline.TestNode2PipelineMap
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/2/artifact/out/Dockerfile
GITHUB PR	#1073
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8cbace0aa0a3 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / d21eccf
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/2/testReport/
Max. process+thread count	5408 (vs. ulimit of 5500)
modules	C: hadoop-hdds/client hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

adoroszlai

Hi @bshashikant,

One of the tests in TestFailureHandlingByClient runs into error with the patch:

[ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 202.465 s <<< FAILURE! - in org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient
[ERROR] testContainerExclusionWithClosedContainerException(org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient)  Time elapsed: 9.579 s  <<< ERROR!
java.lang.IllegalArgumentException
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
	at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:150)
	at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClientForReadData(XceiverClientManager.java:143)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:154)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222)
	at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171)
	at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)

Can you please check? Thanks.

(Note that TestFailureHandlingByClient is not run by the unit CI check, since it's part of the integration test.)

adoroszlai · 2019-07-16T16:20:07Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java

@@ -160,7 +159,7 @@ public BlockOutputStream(BlockID blockID,
    bufferList = null;
    totalDataFlushedLength = 0;
    writtenDataLength = 0;
-    failedServers = Collections.emptyList();
+    failedServers = new ArrayList<>(0);


mukul1987 · 2019-07-16T17:44:07Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java

@@ -284,7 +284,7 @@ private XceiverClientReply sendCommandWithRetry(
        }
        break;
      } catch (ExecutionException | InterruptedException | IOException e) {
-        LOG.debug("Failed to execute command " + request + " on datanode " + dn
+        LOG.info("Failed to execute command " + request + " on datanode " + dn


lets change this to error.

Addressed in the latest patch.

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java

bshashikant · 2019-07-18T04:53:50Z

@adoroszlai , the test failure pointed is tracked by HDDS-1804 separately as the root cause for intermittent failure is same.

hadoop-yetus · 2019-07-18T06:53:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	69	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	20	Maven dependency ordering for branch
+1	mvninstall	462	trunk passed
+1	compile	249	trunk passed
+1	checkstyle	69	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	894	branch has no errors when building and testing our client artifacts.
+1	javadoc	155	trunk passed
0	spotbugs	306	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	495	trunk passed
-0	patch	348	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
0	mvndep	31	Maven dependency ordering for patch
+1	mvninstall	424	the patch passed
+1	compile	256	the patch passed
+1	javac	256	the patch passed
+1	checkstyle	74	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	730	patch has no errors when building and testing our client artifacts.
+1	javadoc	169	the patch passed
+1	findbugs	574	the patch passed
		_ Other Tests _
-1	unit	336	hadoop-hdds in the patch failed.
-1	unit	2014	hadoop-ozone in the patch failed.
+1	asflicense	43	The patch does not generate ASF License warnings.
		7226

Reason	Tests
Failed junit tests	hadoop.hdds.scm.container.placement.algorithms.TestContainerPlacementFactory
	hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
	hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.client.rpc.TestBCSID
	hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
	hadoop.ozone.client.rpc.TestFailureHandlingByClient
	hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
	hadoop.ozone.container.server.TestSecureContainerServer

Subsystem	Report/Notes
Docker	Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/3/artifact/out/Dockerfile
GITHUB PR	#1073
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 4fe98145f0c4 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3dc256e`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/3/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/3/testReport/
Max. process+thread count	5297 (vs. ulimit of 5500)
modules	C: hadoop-hdds/client hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1073/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

mukul1987

+1, the latest changes look good to me.

…y Shashikant Banerjee. (#1073)

bshashikant · 2019-07-18T10:33:54Z

Thanks @mukul1987 @adoroszlai and @supratimdeka for the review. I have committed this change to trunk.

SAMZA-2240:move samza-test to kafka 2.11-0.11

…y Shashikant Banerjee. (apache#1073)

HDDS-1780. TestFailureHandlingByClient tests are flaky.

6cfd24c

bshashikant added the ozone label Jul 10, 2019

bshashikant requested a review from mukul1987 July 10, 2019 13:12

supratimdeka approved these changes Jul 12, 2019

View reviewed changes

Turned off the newtork toplogy switch to ensure tests won't fail.

2536050

Merge branch 'trunk' into HDDS-1780

4b32242

adoroszlai reviewed Jul 16, 2019

View reviewed changes

mukul1987 requested changes Jul 16, 2019

View reviewed changes

Addressed Review Comments.

0d26242

mukul1987 approved these changes Jul 18, 2019

View reviewed changes

bshashikant closed this Jul 18, 2019

bshashikant deleted the HDDS-1780 branch July 18, 2019 10:32

asfgit pushed a commit that referenced this pull request Jul 18, 2019

HDDS-1780. TestFailureHandlingByClient tests are flaky. Contributed b…

ccceedb

…y Shashikant Banerjee. (#1073)

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019

Merge pull request apache#1073 from apache/upgradeToKafka22

5b55d33

SAMZA-2240:move samza-test to kafka 2.11-0.11

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-1780. TestFailureHandlingByClient tests are flaky. Contributed b…

7eb4d1f

…y Shashikant Banerjee. (apache#1073)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-1780. TestFailureHandlingByClient tests are flaky. #1073

HDDS-1780. TestFailureHandlingByClient tests are flaky. #1073

Uh oh!

bshashikant commented Jul 10, 2019

Uh oh!

supratimdeka left a comment

Uh oh!

hadoop-yetus commented Jul 16, 2019

Uh oh!

bshashikant commented Jul 16, 2019

Uh oh!

hadoop-yetus commented Jul 16, 2019

Uh oh!

adoroszlai left a comment

Uh oh!

adoroszlai Jul 16, 2019

Uh oh!

mukul1987 Jul 16, 2019

Uh oh!

bshashikant Jul 18, 2019

Uh oh!

Uh oh!

bshashikant commented Jul 18, 2019

Uh oh!

hadoop-yetus commented Jul 18, 2019

Uh oh!

mukul1987 left a comment

Uh oh!

bshashikant commented Jul 18, 2019

Uh oh!

Uh oh!

HDDS-1780. TestFailureHandlingByClient tests are flaky. #1073

HDDS-1780. TestFailureHandlingByClient tests are flaky. #1073

Uh oh!

Conversation

bshashikant commented Jul 10, 2019

Uh oh!

supratimdeka left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jul 16, 2019

Uh oh!

bshashikant commented Jul 16, 2019

Uh oh!

hadoop-yetus commented Jul 16, 2019

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jul 16, 2019

Choose a reason for hiding this comment

Uh oh!

mukul1987 Jul 16, 2019

Choose a reason for hiding this comment

Uh oh!

bshashikant Jul 18, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bshashikant commented Jul 18, 2019

Uh oh!

hadoop-yetus commented Jul 18, 2019

Uh oh!

mukul1987 left a comment

Choose a reason for hiding this comment

Uh oh!

bshashikant commented Jul 18, 2019

Uh oh!

Uh oh!