[CI] SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions Failure

[talevy]

REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner \
  -Dtests.seed=9861B17A9382A49D \
  -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT \
  -Dtests.method="testSearchTransformInsufficientPermissions" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-Latn-ME \
  -Dtests.timezone=America/Cambridge_Bay \
  -Dcompiler.java=10 \
  -Druntime.java=8

stacktrace

3:45:09 FAILURE 12.0s | SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions <<< FAILURES!
23:45:09    > Throwable #1: java.lang.AssertionError: 
23:45:09    > Expected: is a value equal to or greater than <1>
23:45:09    >      but: <0> was less than <1>
23:45:09    > 	at __randomizedtesting.SeedInfo.seed([9861B17A9382A49D:514B584B289A370]:0)
23:45:09    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
23:45:09    > 	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$3(SmokeTestWatcherWithSecurityIT.java:326)
23:45:09    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:847)
23:45:09    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:821)
...
23:45:30 org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/bulk[s]] is unauthorized for user [watcher_manager]

[elasticmachine]

Pinging @elastic/es-core-infra

[andrershov]

Occurred again https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=centos/58/console

REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner \
  -Dtests.seed=484163AFA937AD5E \
  -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT \
  -Dtests.method="testSearchTransformInsufficientPermissions" \
  -Dtests.security.manager=true \
  -Dtests.locale=it \
  -Dtests.timezone=Antarctica/Mawson \
  -Dcompiler.java=11 \
  -Druntime.java=8

[jakelandis]

I haven't reproduced this exact issue... but stepping through the code, there appears to be a bug in the start/stop logic in SmokeTestWatcherWithSecurityIT such that it does not busy wait as (I believe) is the intent of the code.

Starting Watcher here is wrapped in an assertBusy() which appears to attempt to wait for 10 seconds max until the Watcher is indeed started. However, the break here allows not only to break the switch, but also effectively stops the busy wait logic since it exits the lamda normally. assertBusy only busy waits while an assertionError is getting thrown in the (lamda) code block.

Stop Watcher has the same bug here (note the assertion in both is asserting the success/failure of the http call, not the state of Watcher)... and it looks like SmokeTestWatcherWithSecurityClientYamlTestSuiteIT may have the same issue (but haven't stepped trough that yet).

This means that Watcher is not guaranteed to be started or stopped between tests since it is not properly blocking till started or stopped, and is likely the root cause of many of these Watcher failures.

I believe the fix is to simply change the break to a AssertionError for the state change switches.

Also, this specific test is peculiar since it can pass without Watcher ever started.