[CI] TransportMonitoringMigrateAlertsActionTests.testLocalAlertsRemoval

[andreidan]

Encountered on a PR build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-2/15205/ (multiple tests failing testLocalExporterWithAlertingDisabled, testLocalAlertsRemoval and testDisabledLocalExporterAlertsRemoval

Build scan:
https://gradle-enterprise.elastic.co/s/or4i6aw2kxx6u

Repro line:

./gradlew ':x-pack:plugin:monitoring:test' --tests "org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests.testLocalExporterWithAlertingDisabled" -Dtests.seed=74901503919A9EEE -Dtests.security.manager=true -Dtests.locale=el-GR -Dtests.timezone=Pacific/Pitcairn -Druntime.java=8

Reproduces locally?: No

Applicable branches: 7.11

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.tags=CI&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests&tests.sortField=FAILED&tests.test=testLocalAlertsRemoval&tests.unstableOnly=true

Failure excerpt:

Expected: is <true>
     but: was <false>
	at __randomizedtesting.SeedInfo.seed([74901503919A9EEE:45FE8D342B81AB19]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.monitoring.test.MonitoringIntegTestCase.assertIndicesExists(MonitoringIntegTestCase.java:209)
	at org.elasticsearch.xpack.monitoring.test.MonitoringIntegTestCase.lambda$awaitIndexExists$3(MonitoringIntegTestCase.java:204)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1016)
	at org.elasticsearch.xpack.monitoring.test.MonitoringIntegTestCase.awaitIndexExists(MonitoringIntegTestCase.java:204)
	at org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests.waitForWatcherIndices(TransportMonitoringMigrateAlertsActionTests.java:592)
	at org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests.ensureInitialLocalResources(TransportMonitoringMigrateAlertsActionTests.java:473)
	at org.elasticsearch.xpack.monitoring.action.TransportMonitoringMigrateAlertsActionTests.testLocalAlertsRemoval(TransportMonitoringMigrateAlertsActionTests.java:139)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[gwbrown]

These have been failing more recently on master, 7.x, and 7.11 so I'm going to mute these pending a fix.

Some possibly-relevant logs from the failures:

[2020-12-18T08:15:12,687][ERROR][o.e.x.c.t.IndexTemplateRegistry] [node_s1] error adding index template [.watch-history-14] from [/watch-history.json] for [watcher]	
java.lang.IllegalArgumentException: composable template [.watch-history-14] template after composition is invalid	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:504) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:399) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:59) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:680) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]	
	at java.lang.Thread.run(Thread.java:834) [?:?]	
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [index_template] unknown field [data_stream]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.lambda$validateCompositeTemplate$21(MetadataIndexTemplateService.java:1133) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:638) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.validateCompositeTemplate(MetadataIndexTemplateService.java:1113) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:500) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	... 14 more	
[2020-12-18T08:15:12,690][ERROR][o.e.x.c.t.IndexTemplateRegistry] [node_s1] error adding index template [.slm-history] from [/slm-history.json] for [index_lifecycle]	
java.lang.IllegalArgumentException: composable template [.slm-history] template after composition is invalid	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:504) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:399) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:59) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:680) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]	
	at java.lang.Thread.run(Thread.java:834) [?:?]	
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [index_template] unknown field [data_stream]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.lambda$validateCompositeTemplate$21(MetadataIndexTemplateService.java:1133) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:638) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.validateCompositeTemplate(MetadataIndexTemplateService.java:1113) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:500) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]	
	... 14 more

[martijnvg]

The mentioned stacktrace is caused by the fact that the monitoring test doesn't load the data stream plugin and then the watcher history index template can't be added because it uses data stream (the data stream field type is missing). I also noticed that other plugins are not loaded which monitoring requires to run (enrich for EnrichStatsCollector and cc for StatsCollector). I will fix this and that should reduce the noice.