[CI] Failure in TimeThrottleIntegrationTests.testTimeThrottle

[elasticmachine]

Original comment by @albertzaharovits:

Failure is in this build: LINK REDACTED
Repro cmd which is not working locally is:

./gradlew :x-pack-elasticsearch:plugin:watcher:test \
  -Dtests.seed=FE582B72FA7C67FE \
  -Dtests.class=org.elasticsearch.xpack.watcher.actions.TimeThrottleIntegrationTests \
  -Dtests.method="testTimeThrottle" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-NI \
  -Dtests.timezone=Asia/Khandyga

The symptom is:

1:32:30    > Throwable LINK REDACTED: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 	at __randomizedtesting.SeedInfo.seed([FE582B72FA7C67FE:10F46257D65AAB07]:0)
11:32:30    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:734)
11:32:30    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:708)
11:32:30    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.startWatcher(AbstractWatcherIntegrationTestCase.java:465)
11:32:30    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase._setup(AbstractWatcherIntegrationTestCase.java:207)
11:32:30    > 	at java.lang.Thread.run(Thread.java:748)
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STOPPED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTING], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STARTING>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more
11:32:30    > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s1, v2=STARTED], Tuple [v1=node_s0, v2=STOPPED]]
11:32:30    > Expected: every item is is <STARTED>
11:32:30    >      but: an item was <STOPPED>
11:32:30    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:32:30    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:479)
11:32:30    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:722)
11:32:30    > 		... 39 more

This is similar to others, such as LINK REDACTED and LINK REDACTED so presumably fixing the underlying cause will solve some of them at least.

[elasticmachine]

Original comment by @javanna:

A different test, but exactly the same symptoms:
LINK REDACTED

REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:plugin:watcher:test \
  -Dtests.seed=B7FFE970B7ADF3AF \
  -Dtests.class=org.elasticsearch.xpack.watcher.actions.webhook.WebhookHttpsIntegrationTests \
  -Dtests.method="testHttpsAndBasicAuth" \
  -Dtests.security.manager=true \
  -Dtests.locale=de-AT \
  -Dtests.timezone=Europe/Tirane

REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:plugin:watcher:test \
  -Dtests.seed=B7FFE970B7ADF3AF \
  -Dtests.class=org.elasticsearch.xpack.watcher.actions.webhook.WebhookHttpsIntegrationTests \
  -Dtests.method="testHttpsAndBasicAuth" \
  -Dtests.security.manager=true \
  -Dtests.locale=de-AT \
  -Dtests.timezone=Europe/Tirane

@spinscale I went for posting it here instead of opening another issue, let me know if I should raise another one.

[elasticmachine]

Original comment by @spinscale:

all good, I opened LINK REDACTED for this

[elasticmachine]

Original comment by @dimitris-athanasiou:

This seems somewhat related as well: LINK REDACTED

REPRODUCE WITH: ./gradlew :x-pack-elasticsearch:qa:smoke-test-watcher-with-security:integTestRunner -Dtests.seed=301BC8944397CEE3 -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT -Dtests.method="testSearchInputHasPermissions" -Dtests.security.manager=true -Dtests.locale=es-DO -Dtests.timezone=Asia/Baku

Fails with:

Throwable LINK REDACTED: java.lang.AssertionError: 
09:01:15    > Expected: is "started"
09:01:15    >      but: was "stopped"
09:01:15    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
09:01:15    > 	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$startWatcher$0(SmokeTestWatcherWithSecurityIT.java:74)
09:01:15    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:731)
09:01:15    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
09:01:15    > 	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.startWatcher(SmokeTestWatcherWithSecurityIT.java:63)
09:01:15    > 	at java.lang.Thread.run(Thread.java:748)
09:01:15    > 	Suppressed: java.lang.AssertionError: 
09:01:15    > Expected: is "started"
09:01:15    >      but: was "stopping"
09:01:15    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
09:01:15    > 		at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$startWatcher$0(SmokeTestWatcherWithSecurityIT.java:74)
09:01:15    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
09:01:15    > 		... 38 more

[elasticmachine]

Original comment by @dimitris-athanasiou:

More tests failing due to start/stop problems: LINK REDACTED

[elasticmachine]

Original comment by @romseygeek:

This also looks like it's due to Watcher start/stop issues: LINK REDACTED

[jasontedor]

Closed by #30118