Meta: Fix Watcher Test Failures

[jakelandis]

The number of Watcher test failures which have been muted is too high. This is a Meta issue to help track and resolve those test failures.

https://github.com/elastic/elasticsearch/issues?utf8=%E2%9C%93&q=is%3Aopen+label%3A%3ACore%2FFeatures%2FWatcher+label%3A%3Etest-failure

WatcherRestIT

• Failure: WatcherRestIT mustache/50_webhook_url_escaping/Test url escaping with url mustache function failure #41172
• Un-mute + debug logging: unmute 'Test url escaping with url mustache function' and bump logging #42400 (master only)
• Failure [CI] failure WatcherRestIT yaml=painless/20_minimal_body/Test execute watch api with minimal body #43988, muted on master and 7.x

Rolling Upgrade Tests

• Failure: Watcher: Mysterious rolling upgrade failure #33185
• Un-mute + debug logging: un-mute Watcher rolling upgrade tests and bump up logging #42377 (master only)

SmokeTestWatcherWithSecurityIT

• Failures: CI: SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions fails #35361, [CI] SmokeTestWatcherWithSecurityIT suite failures #30777, CI: SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions fails #35361, [CI] SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions Failure #33291, [CI] SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges Failure #29893
• Un-mute + debug logging: Address test failures for SmokeTestWatcherWithSecurityIT #42092 (master only)
• [CI] SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges Failure #29893 fixed by address SmokeTestWatcherWithSecurityIT#testSearchInputWithInsufficientPrivileges #42764
• SmokeTestWatcherWithSecurityIT: Retry if failures searching .watcher-history #43781 may address these failures

WatchBackwardsCompatibilityIT

• Failure: [CI] WatchBackwardsCompatibilityIT failures #40841
• Fixed by: Do not use ifSeqNo for update requests on mixed cluster #42596
• Failure WatchBackwardsCompatibilityIT failure #44814

SingleNodeTests

• Failure: [CI] Watcher SingleNodeTests failed due to running threads #36782
• Un-mute + debug logging: TODO

BasicWatcherTests

• Failure: [CI] BasicWatcherTests.testModifyWatches fails in CI #35503
• Un-mute + debug logging: TODO

SmokeTestWatcherTestSuiteIT

• Failures: [ci] SmokeTestWatcherTestSuiteIT.testMonitorClusterHealth #32299, [CI] SmokeTestWatcherWithSecurityIT.* failing #29877
• Un-mute + debug logging: TODO

EmailAttachmentTests

• Failure: testThatEmailAttachmentsAreSent failed due to internal cluster not forming #30870
• Un-mute + debug logging: TODO

ActivateWatchTests

• Failure: [CI] ActivateWatchTests.testDeactivateAndActivate needs to not be sleep based #30699
• Un-mute + debug logging: un-mute ActivateWatchTests, bump up logging, and remove explicit sleeps #42396 (master only)

Template to add to test failure issues if/when a test fails.

Un-muted this test on PR #____ to obtain additional logs. 

If (when?) this test fails again please obtain the following information before muting the test:

* Copy of the relevant failure
* Copy of the reproduce line
* The Jenkins build link 
* The Gradle scan link
* The relevant cluster logs from "Google Cloud Storage Upload Report" (link found in Jenkins build)

The tests will only be un-muted on master and if no failures after a certain period of time, will back port the un-mute.

[elasticmachine]

Pinging @elastic/es-core-features

[jakelandis]

All tests in SmokeTestWatcherWithSecurityIT failed. It is likely that all share the same root cause.

Test Result (6 failures / +6)
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions
    org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions

Relevant Failure with repro lines:

10:45:18 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchInputWithInsufficientPrivileges FAILED
10:45:18     org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45978], URI [/.watcher-history-*/_search?rest_total_hits_as_int=true], status line [HTTP/1.1 503 Service Unavailable]
10:45:18     {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
10:45:18         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:959B3C83AD12D929]:0)
10:45:18         at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
10:45:18         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
10:45:18         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:326)
10:45:18         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:850)
10:45:18         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:45:18         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:182)
10:45:29 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:45:29 
10:45:29 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testIndexActionHasPermissions FAILED
10:45:29     java.lang.AssertionError: 
10:45:29     Expected: is a value equal to or greater than <1>
10:45:29          but: <0> was less than <1>
10:45:29         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:1F46402E52F9591E]:0)
10:45:29         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:45:29         at org.junit.Assert.assertThat(Assert.java:956)
10:45:29         at org.junit.Assert.assertThat(Assert.java:923)
10:45:29         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:45:29         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:45:29         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:29         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:29         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions(SmokeTestWatcherWithSecurityIT.java:259)
10:45:40 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:45:40 
10:45:40 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testIndexActionInsufficientPrivileges FAILED
10:45:40     java.lang.AssertionError: 
10:45:40     Expected: is a value equal to or greater than <1>
10:45:40          but: <0> was less than <1>
10:45:40         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:204F346B78395B47]:0)
10:45:40         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:45:40         at org.junit.Assert.assertThat(Assert.java:956)
10:45:40         at org.junit.Assert.assertThat(Assert.java:923)
10:45:40         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:45:40         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:45:40         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:40         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:40         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:282)
10:45:51 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:45:51 
10:45:51 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchTransformHasPermissions FAILED
10:45:51     java.lang.AssertionError: 
10:45:51     Expected: is a value equal to or greater than <1>
10:45:51          but: <0> was less than <1>
10:45:51         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:138387687DCC68CA]:0)
10:45:51         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:45:51         at org.junit.Assert.assertThat(Assert.java:956)
10:45:51         at org.junit.Assert.assertThat(Assert.java:923)
10:45:51         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:45:51         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:45:51         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:45:51         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:45:51         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions(SmokeTestWatcherWithSecurityIT.java:210)
10:46:01 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:01 
10:46:01 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchTransformInsufficientPermissions FAILED
10:46:01     java.lang.AssertionError: 
10:46:01     Expected: is a value equal to or greater than <1>
10:46:01          but: <0> was less than <1>
10:46:01         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:A786648885C87DCA]:0)
10:46:01         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:01         at org.junit.Assert.assertThat(Assert.java:956)
10:46:01         at org.junit.Assert.assertThat(Assert.java:923)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:01         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:01         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:46:01         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions(SmokeTestWatcherWithSecurityIT.java:239)
10:46:12 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12 
10:46:12 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT > testSearchInputHasPermissions FAILED
10:46:12     java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:4822DFE1A9577852]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions(SmokeTestWatcherWithSecurityIT.java:158)
10:46:12 REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12 
10:46:12 Suite: Test class org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT
10:46:12   1> [2019-06-06T21:45:13,833][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] before test
10:46:12   1> [2019-06-06T21:45:13,838][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] initializing REST clients against [http://[::1]:45978]
10:46:12   1> [2019-06-06T21:45:18,712][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:18,713][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputWithInsufficientPrivileges] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45978], URI [/.watcher-history-*/_search?rest_total_hits_as_int=true], status line [HTTP/1.1 503 Service Unavailable]
10:46:12     {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:959B3C83AD12D929]:0)
10:46:12         at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
10:46:12         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
10:46:12         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:326)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:850)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:182)
10:46:12   1> [2019-06-06T21:45:18,748][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionHasPermissions] before test
10:46:12   1> [2019-06-06T21:45:29,504][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionHasPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:29,505][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionHasPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:1F46402E52F9591E]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions(SmokeTestWatcherWithSecurityIT.java:259)
10:46:12   1> [2019-06-06T21:45:29,532][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionInsufficientPrivileges] before test
10:46:12   1> [2019-06-06T21:45:40,309][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionInsufficientPrivileges] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:40,310][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testIndexActionInsufficientPrivileges] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:204F346B78395B47]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:282)
10:46:12   1> [2019-06-06T21:45:40,331][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformHasPermissions] before test
10:46:12   1> [2019-06-06T21:45:51,016][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformHasPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:45:51,017][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformHasPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:138387687DCC68CA]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions(SmokeTestWatcherWithSecurityIT.java:210)
10:46:12   1> [2019-06-06T21:45:51,041][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformInsufficientPermissions] before test
10:46:12   1> [2019-06-06T21:46:01,747][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformInsufficientPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:46:01,747][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchTransformInsufficientPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:A786648885C87DCA]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions(SmokeTestWatcherWithSecurityIT.java:239)
10:46:12   1> [2019-06-06T21:46:01,768][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputHasPermissions] before test
10:46:12   1> [2019-06-06T21:46:12,464][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputHasPermissions] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk, indices:data/write/bulk[s]].
10:46:12   1> [2019-06-06T21:46:12,465][INFO ][o.e.s.SmokeTestWatcherWithSecurityIT] [testSearchInputHasPermissions] after test
10:46:12   2> REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions" -Dtests.seed=3AF36076A4C37A27 -Dtests.security.manager=true -Dtests.locale=nl-BQ -Dtests.timezone=Asia/Bishkek -Dcompiler.java=12 -Druntime.java=11
10:46:12   2> java.lang.AssertionError: 
10:46:12     Expected: is a value equal to or greater than <1>
10:46:12          but: <0> was less than <1>
10:46:12         at __randomizedtesting.SeedInfo.seed([3AF36076A4C37A27:4822DFE1A9577852]:0)
10:46:12         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
10:46:12         at org.junit.Assert.assertThat(Assert.java:956)
10:46:12         at org.junit.Assert.assertThat(Assert.java:923)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:329)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
10:46:12         at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
10:46:12         at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions(SmokeTestWatcherWithSecurityIT.java:158)
10:46:12   2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/x-pack/qa/smoke-test-watcher-with-security/build/testrun/integTestRunner/temp/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT_3AF36076A4C37A27-001
10:46:12   2> NOTE: test params are: codec=Lucene80, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@53309619), locale=nl-BQ, timezone=Asia/Bishkek
10:46:12   2> NOTE: Linux 3.10.0-957.12.2.el7.x86_64 amd64/Oracle Corporation 11.0.3 (64-bit)/cpus=16,threads=1,free=426557952,total=536870912
10:46:12   2> NOTE: All tests run in this JVM: [SmokeTestWatcherWithSecurityClientYamlTestSuiteIT, SmokeTestWatcherWithSecurityIT]
10:46:12 
10:46:12 66 tests completed, 6 failed
10:46:12 
10:46:12 Tests with failures:
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputWithInsufficientPrivileges
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionHasPermissions
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions
10:46:12  - org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions

Jenkins build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/37
Gradle build: https://scans.gradle.com/s/ab6f6mtnx5wvq
Logs: https://storage.cloud.google.com/elasticsearch-ci-artifacts/jobs/elastic+elasticsearch+master+intake/37/log-files.tar.gz
Relevant logs: relevant_logs.tar.gz

[droberts195]

In https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/110/console just one test failed, but it's the same as one of the failures in #42409 (comment) and failed with the exact same reason as the first failure in #42409 (comment).

The failure is:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43041], URI [/.watcher-history-*/_search?rest_total_hits_as_int=true], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
	at __randomizedtesting.SeedInfo.seed([73CA3AF4F90C540D:69766EE925F6756D]:0)
	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$4(SmokeTestWatcherWithSecurityIT.java:326)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:850)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306)
	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testIndexActionInsufficientPrivileges(SmokeTestWatcherWithSecurityIT.java:282)

At the time this happened the corresponding error in the server side log was:

[2019-06-11T13:59:08,130][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [my_test_index] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-06-11T13:59:08,593][INFO ][o.e.c.m.MetaDataMappingService] [node-0] [my_test_index/MWObuPVBTcC-nxbj8SRzUQ] create_mapping [_doc]
[2019-06-11T13:59:08,837][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [index_not_allowed_to_read] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-06-11T13:59:08,981][INFO ][o.e.c.m.MetaDataMappingService] [node-0] [index_not_allowed_to_read/t8g7ay5HTrGHyDj0TAgzAg] create_mapping [_doc]
[2019-06-11T13:59:09,232][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.watches] creating index, cause [auto(bulk api)], templates [.watches], shards [1]/[0], mappings [_doc]
[2019-06-11T13:59:09,370][INFO ][o.e.x.w.WatcherService   ] [node-0] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
[2019-06-11T13:59:09,392][DEBUG][o.e.x.w.WatcherIndexingListener] [node-0] adding watch [FbxatmpquAmnjMGcegxg] to trigger service
[2019-06-11T13:59:09,473][DEBUG][o.e.x.w.WatcherService   ] [node-0] watch service has been reloaded, reason [new local watcher shard allocation ids]
[2019-06-11T13:59:10,502][DEBUG][o.e.x.w.t.s.e.TickerScheduleTriggerEngine] [node-0] triggered job [FbxatmpquAmnjMGcegxg] at [2019-06-11T13:59:10.502Z] (scheduled time was [2019-06-11T13:59:10.394Z])
[2019-06-11T13:59:10,572][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.triggered_watches] creating index, cause [auto(bulk api)], templates [.triggered_watches], shards [1]/[1], mappings [_doc]
[2019-06-11T13:59:10,576][INFO ][o.e.c.r.a.AllocationService] [node-0] updating number_of_replicas to [0] for indices [.triggered_watches]
[2019-06-11T13:59:11,560][DEBUG][o.e.x.w.t.s.e.TickerScheduleTriggerEngine] [node-0] triggered job [FbxatmpquAmnjMGcegxg] at [2019-06-11T13:59:11.560Z] (scheduled time was [2019-06-11T13:59:11.394Z])
[2019-06-11T13:59:11,607][DEBUG][o.e.x.w.e.ExecutionService] [node-0] executing watch [FbxatmpquAmnjMGcegxg]
[2019-06-11T13:59:11,643][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.watcher-history-9-2019.06.11] creating index, cause [auto(bulk api)], templates [.watch-history-no-ilm-9], shards [1]/[0], mappings [doc]
[2019-06-11T13:59:11,630][ERROR][o.e.x.w.a.i.IndexActionFactory] [node-0] failed to execute action [FbxatmpquAmnjMGcegxg/index]
org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/bulk[s]] is unauthorized for user [watcher_manager]
...
...very long stack trace...
...
[2019-06-11T13:59:11,703][DEBUG][o.e.x.w.WatcherIndexingListener] [node-0] adding watch [FbxatmpquAmnjMGcegxg] to trigger service
[2019-06-11T13:59:11,735][DEBUG][o.e.a.s.TransportSearchAction] [node-0] All shards failed for phase: [query]
[2019-06-11T13:59:11,737][WARN ][r.suppressed             ] [node-0] path: /.watcher-history-*/_search, params: {rest_total_hits_as_int=true, index=.watcher-history-*}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1101) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1213) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1187) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-06-11T13:59:11,802][INFO ][o.e.x.w.WatcherService   ] [node-0] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
[2019-06-11T13:59:11,856][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [my_test_index/MWObuPVBTcC-nxbj8SRzUQ] deleting index
[2019-06-11T13:59:11,987][INFO ][o.e.c.m.MetaDataMappingService] [node-0] [.watcher-history-9-2019.06.11/uLPXpq2YSmadsCt7RdBAoQ] update_mapping [doc]

[alpar-t]

I'we been working on running the build with more parallelism enabled which has unfortunately caused more (I believe) timing related failures such as this one: https://scans.gradle.com/s/m44sa6oyftyue/tests/6ej7h6toy4af6-6g5rmtb4agtbo

[hendrikmuhs]

more failures:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux-6/95/consoleFull