[ci] XPackIT.testIndexTemplatesCreated in :qa:rolling-upgrade

[andyb-elastic]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+default-distro/181/console

CI log
build-181.txt

Integ test cluster logs

• v5.6.11-SNAPSHOT#oldClusterTestCluster node 0 run.log
• v5.6.11-SNAPSHOT#oldClusterTestCluster node 1 run.log
• v5.6.11-SNAPSHOT#oldClusterTestCluster node 2 run.log
• v5.6.11-SNAPSHOT#oneThirdUpgradedTestCluster node 0 run.log

./gradlew :qa:rolling-upgrade:v5.6.11-SNAPSHOT#oneThirdUpgradedTestRunner \
  -Dtests.seed=C90DE3BF10138E76 \
  -Dtests.class=org.elasticsearch.upgrades.XPackIT \
  -Dtests.method="testIndexTemplatesCreated" \
  -Dtests.security.manager=true \
  -Dtests.locale=ja-JP-u-ca-japanese-x-lvariant-JP \
  -Dtests.timezone=US/Eastern \
  -Dtests.distribution=zip

FAILURE 10.1s | XPackIT.testIndexTemplatesCreated <<< FAILURES!                               
   > Throwable #1: java.lang.AssertionError: expected:<[.triggered_watches, .watch-history-8, .watches]> but was:<[.triggered_watches, .watch-history-9, .watches]>                             >    at __randomizedtesting.SeedInfo.seed([C90DE3BF10138E76:DF95C5B3FFA37179]:0)           
   >    at org.elasticsearch.upgrades.XPackIT.lambda$testIndexTemplatesCreated$0(XPackIT.java:129)                                                                                           
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:815)                  
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:789)                  
   >    at org.elasticsearch.upgrades.XPackIT.testIndexTemplatesCreated(XPackIT.java:111)     
   >    at java.lang.Thread.run(Thread.java:748)   

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

I see a Watcher template as the offender here. Would @spinscale or @hub-cap take an initial look?

[spinscale]

Opened #32352 to fix this, an old template history template is referred...

[spinscale]

closed by #32352