[CI] ':qa:mixed-cluster:v7.X.X#mixedClusterTest' failures

[iverase]

There are a few test using mixed clusters that are failing. I was able to reproduce locally using the following:

./gradlew ':qa:mixed-cluster:v7.7.0#mixedClusterTest' --tests "org.elasticsearch.backwards.IndexingIT"   -Dtests.seed=4C0785959C950BAF   -Dtests.security.manager=true   -Dtests.locale=he   -Dtests.timezone=America/Barbados   -Dcompiler.java=14

The error that seems to be causing this errors is:

[[reason=ALLOCATION_FAILED], at[2020-04-21T14:24:17.813Z], failed_attempts[1], failed_nodes[[FRHbNZRZTQarwsO04vHPIw]], delayed=false, details[failed shard on node [FRHbNZRZTQarwsO04vHPIw]: failed to create index, failure IllegalArgumentException[unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]], allocation_status[no_attempt]], message [failed to create index], failure [IllegalArgumentException[unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]], markAsStale [true]]
»  java.lang.IllegalArgumentException: unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
»       at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:595) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:549) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:176) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:484) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:246) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]
»       at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) ~[elasticsearch-7.8.0-SNAPSHOT.jar:7.8.0-SNAPSHOT]

Looking in the repo it seems to be related to this change (which is a breaking change): #55411

examples:

https://gradle-enterprise.elastic.co/s/6ovzg4cjhkab4
https://gradle-enterprise.elastic.co/s/pv5ozkxb3nlxm

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

[dakrone]

@iverase thanks for reporting this, it was a real issue. I opened a PR to fix it.

[dakrone]

This is resolved in #55546 but since it was a 7.x-only PR it didn't automatically close, so I'm closing it now.