HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase #1757

liuml07 · 2019-12-12T02:22:35Z

https://issues.apache.org/jira/browse/HADOOP-16757

…taStore

liuml07 · 2019-12-12T02:24:43Z

@steveloughran Does this sound good?

Thanks,

hadoop-yetus · 2019-12-12T03:32:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	4m 25s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	18m 45s	trunk passed
+1 💚	compile	0m 33s	trunk passed
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 38s	trunk passed
+1 💚	shadedclient	13m 46s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 26s	trunk passed
+0 🆗	spotbugs	0m 59s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 58s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s	the patch passed
+1 💚	compile	0m 26s	the patch passed
+1 💚	javac	0m 26s	the patch passed
+1 💚	checkstyle	0m 18s	the patch passed
+1 💚	mvnsite	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	12m 45s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	the patch passed
+1 💚	findbugs	1m 3s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 24s	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 30s	The patch does not generate ASF License warnings.
		59m 11s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/2/artifact/out/Dockerfile
GITHUB PR	#1757
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 3c8f2fd74c84 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `93bb368`
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/2/testReport/
Max. process+thread count	451 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-12-12T03:37:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 15s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 40s	trunk passed
+1 💚	compile	0m 36s	trunk passed
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 37s	trunk passed
+1 💚	shadedclient	14m 49s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	trunk passed
+0 🆗	spotbugs	0m 59s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 57s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s	the patch passed
+1 💚	compile	0m 26s	the patch passed
+1 💚	javac	0m 26s	the patch passed
+1 💚	checkstyle	0m 17s	the patch passed
+1 💚	mvnsite	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	16m 0s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 26s	the patch passed
+1 💚	findbugs	1m 31s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 39s	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 33s	The patch does not generate ASF License warnings.
		64m 8s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/1/artifact/out/Dockerfile
GITHUB PR	#1757
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e0ce5fe03305 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `93bb368`
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/1/testReport/
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-12-12T18:14:46Z

+1

thanks

liuml07 · 2019-12-12T23:52:38Z

Since this time out is failing us by being small only sometimes, before commit, I tested more. I then found out that one point I have been missing is that: multiple rules are taking effect at the same time.

So I ran that test 100 times, and some times it failed with testAncestorOverwriteConflict...test timed out after 100000 milliseconds. I then realized that: the base class HadoopTestBase has its own rule and it was not suppressed by the rule we newly add in this patch. So two time out rules (one in base class HadoopTestBase and the other newly added here), the smaller wins.

So there are two ways:

Make MetadataStoreTestBase not extend HadoopTestBase. I see other integration tests in fs/s3a are not extending this base class and they are using timeout defined on their own - which was what we expected.
Make the rule in HadoopTestBase overridable in child classes, i.e. making HadoopTestBase::retrieveTestTimeout method non-static (and protected?). Then override it in MetadataStoreTestBase

In new commit of this PR, I'll make use the 2nd approach since other tests may benefit from this changes, and our MetadataStoreTestBase could keep using other good stuff in the base class, e.g. better test name.

@steveloughran Sorry I did not test enough when I filed PR, and fortunately the error showed up when I tested 100 times (not really 100, I killed it when the error happened).

liuml07 · 2019-12-13T00:18:11Z

I found a simple way to test. The following toy test in MetadataStoreTestBase now succeed.

  @Test
  public void testSleep() throws Exception {
    Thread.currentThread().sleep(S3ATestConstants.S3A_TEST_TIMEOUT / 2);
  }

hadoop-yetus · 2019-12-13T02:24:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 59s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 4s	Maven dependency ordering for branch
+1 💚	mvninstall	20m 34s	trunk passed
+1 💚	compile	18m 6s	trunk passed
+1 💚	checkstyle	2m 50s	trunk passed
+1 💚	mvnsite	2m 4s	trunk passed
+1 💚	shadedclient	20m 5s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 53s	trunk passed
+0 🆗	spotbugs	1m 6s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 8s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 21s	the patch passed
+1 💚	compile	17m 21s	the patch passed
+1 💚	javac	17m 21s	the patch passed
+1 💚	checkstyle	2m 49s	the patch passed
+1 💚	mvnsite	2m 4s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 54s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 55s	the patch passed
+1 💚	findbugs	3m 25s	the patch passed
		_ Other Tests _
-1 ❌	unit	8m 49s	hadoop-common in the patch failed.
+1 💚	unit	1m 35s	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 47s	The patch does not generate ASF License warnings.
		125m 54s

Reason	Tests
Failed junit tests	hadoop.security.TestFixKerberosTicketOrder

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/3/artifact/out/Dockerfile
GITHUB PR	#1757
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f1a14b9427c8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 65c4660
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/3/testReport/
Max. process+thread count	1341 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1757/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

liuml07 · 2019-12-13T04:23:18Z

The failing UT seems not relevant and I have file HADOOP-16760

steveloughran · 2019-12-13T12:51:08Z

LGTM

+1

thanks!

…se (apache#1757) Contributed by Mingliang Liu. Signed-off-by: Steve Loughran <stevel@apache.org>

liuml07 added the enhancement label Dec 12, 2019

liuml07 requested a review from steveloughran December 12, 2019 02:22

liuml07 self-assigned this Dec 12, 2019

HADOOP-16757. Increase timeout unit test rule for ITestDynamoDBMetada…

bc530af

…taStore

liuml07 force-pushed the HADOOP-16757 branch from 7b2f227 to bc530af Compare December 12, 2019 02:24

liuml07 added the fs/s3 changes related to hadoop-aws; submitter must declare test endpoint label Dec 12, 2019

Make the rule in HadoopTestBase overridable in child classes

28ce5c2

liuml07 merged commit d12ad9e into apache:trunk Dec 13, 2019

liuml07 deleted the HADOOP-16757 branch December 13, 2019 16:19

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBa…

2371057

…se (apache#1757) Contributed by Mingliang Liu. Signed-off-by: Steve Loughran <stevel@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase #1757

HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase #1757

Uh oh!

liuml07 commented Dec 12, 2019

Uh oh!

liuml07 commented Dec 12, 2019

Uh oh!

hadoop-yetus commented Dec 12, 2019

Uh oh!

hadoop-yetus commented Dec 12, 2019

Uh oh!

steveloughran commented Dec 12, 2019

Uh oh!

liuml07 commented Dec 12, 2019 •

edited

Loading

Uh oh!

liuml07 commented Dec 13, 2019

Uh oh!

hadoop-yetus commented Dec 13, 2019

Uh oh!

liuml07 commented Dec 13, 2019 •

edited

Loading

Uh oh!

steveloughran commented Dec 13, 2019

Uh oh!

Uh oh!

HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase #1757

HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase #1757

Uh oh!

Conversation

liuml07 commented Dec 12, 2019

Uh oh!

liuml07 commented Dec 12, 2019

Uh oh!

hadoop-yetus commented Dec 12, 2019

Uh oh!

hadoop-yetus commented Dec 12, 2019

Uh oh!

steveloughran commented Dec 12, 2019

Uh oh!

liuml07 commented Dec 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liuml07 commented Dec 13, 2019

Uh oh!

hadoop-yetus commented Dec 13, 2019

Uh oh!

liuml07 commented Dec 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Dec 13, 2019

Uh oh!

Uh oh!

liuml07 commented Dec 12, 2019 •

edited

Loading

liuml07 commented Dec 13, 2019 •

edited

Loading