[CI] watcher/usage/10_basic/Test watcher usage stats output fails

[dnhatn]

watcher/usage/10_basic/Test watcher usage stats output fails with an unexpected watch_count_active.

> Caused by: java.lang.AssertionError: field [watcher.count.active] is not greater than [$watch_count_active]
> Expected: a value greater than <0>
>      but: <0> was equal to <0>
>    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)

I can reproduce this locally with this command:

./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner \
  -Dtests.seed=E3C29FF2DA312869 \
  -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=watcher/usage/10_basic/Test watcher usage stats output}" \
  -Dtests.security.manager=true \
  -Dtests.locale=ru \
  -Dtests.timezone=America/Belize \
  -Dcompiler.java=10 \
  -Druntime.java=8

CI instances:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/6956/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/2371/console

[elasticmachine]

Pinging @elastic/es-core-infra

[dakrone]

This failed again today with:

09:55:26 
09:55:26 org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT > test {yaml=watcher/usage/10_basic/Test watcher usage stats output} FAILED
09:55:26     java.lang.AssertionError: Failure at [watcher/usage/10_basic:48]: field [watcher.count.active] is not greater than [$watch_count_active]
09:55:26     Expected: a value greater than <1>
09:55:26          but: <1> was equal to <1>
09:55:26         at __randomizedtesting.SeedInfo.seed([3AA5E1A0040CF825:B2F1DE7AAAF095DD]:0)
09:55:26         at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:405)
09:55:26         at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:382)
09:55:26         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:55:26         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

I was unable to reproduce this on the 7.x branch.

Failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_RUNTIME_JAVA=zulu11,nodes=general-purpose/538/consoleFull
https://gradle-enterprise.elastic.co/s/xxhruxec3jeji

[martijnvg]

Thanks @dakrone for triaging this. This issue is similar to #52453. Several tests have issues with producing the correct watch count via watcher stats api api. The usage api, uses the watcher stats api under the hood. I will close this issue and add this comment to #52453.