[CI] SmokeTestWatcherWithSecurityIT.testSearchInputHasPermissions flaky failures

[talevy]

master and 6.x have been running into many instances of this test failing

here is one reproduce snippet that doesn't always reproduce

REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner \
  -Dtests.seed=7EF35AACC2EC51CB \
  -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT \
  -Dtests.method="testSearchInputHasPermissions" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-Latn-BA \
  -Dtests.timezone=Pacific/Fakaofo \
  -Dcompiler.java=10 \
  -Druntime.java=10

[elasticmachine]

Pinging @elastic/es-core-infra

[Tim-Brooks]

Thanks for doing this. I was just about to mute these tests myself.

[Tim-Brooks]

Failure here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/273/console

[Tim-Brooks]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=virtual&&linux/273/console

[Tim-Brooks]

I also muted testSearchTransformHasPermissions and testIndexActionInsufficientPrivileges because they also appear to be failing.

[talevy]

thanks @tbrooks8! I kept bouncing back and forth between how many to mute. For whatever reason, they are all going to town recently