testWatcherRestart occasionally fails in REST tests

[DaveCTurner]

I count 10 failures of WatchBackwardsCompatibilityIT#testWatcherRestart since June 13th, all in the 6.x series (i.e. 6.2, 6.3 or 6.4 as well as 6.x). They are not all the same as the failure described here, but some of them are. The latest one was in a BWC test but non-BWC tests also failed in this fashion.

The build link is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+bwc-tests/1/console

The reproduction line follows, but does not reproduce for me:

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0#oldClusterTestRunner \
  -Dtests.seed=A6F6BC1F82877D71 \
  -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT \
  -Dtests.method="testWatcherRestart" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-VE \
  -Dtests.timezone=Etc/GMT+4

The output follows.

java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
	at __randomizedtesting.SeedInfo.seed([A6F6BC1F82877D71:33AEB290C99BA428]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:830)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:804)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$12(WatchBackwardsCompatibilityIT.java:297)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:273)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.ensureWatcherStopped(WatchBackwardsCompatibilityIT.java:297)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.testWatcherRestart(WatchBackwardsCompatibilityIT.java:184)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"starting\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"starting\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:301)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"starting\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"starting\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:301)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more

[elasticmachine]

Pinging @elastic/es-core-infra

[dliappis]

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/111/consoleText

[danielmitterdorfer]

Looking at the failure output in the original comment, we see that on one node (node id -34h6dtFQ2SsSYIJ_n0LKw), Watcher is in the state starting and then moves to started over time. So maybe the problem is that before we attempt to stop Watcher in:

elasticsearch/x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/WatchBackwardsCompatibilityIT.java

Line 183 in 230cd10

executeAgainstRandomNode(client -> assertOK(client.performRequest("POST", "/_xpack/watcher/_stop")));

we'd actually need to ensure that all watches are started?

@spinscale / @hub-cap can one of you please have a look at this one?

[andrershov]

Occurred again https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=debian/58/console

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:without-system-key:v6.4.3-SNAPSHOT#oldClusterTestRunner \
  -Dtests.seed=EA5213BE918AF24A \
  -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT \
  -Dtests.method="testWatcherRestart" \
  -Dtests.security.manager=true \
  -Dtests.locale=el \
  -Dtests.timezone=Africa/Lagos \
  -Dcompiler.java=11 \
  -Druntime.java=8