[CI] Frequent failures in MixedClusterClientYamlTestSuiteIT running snapshot yaml tests

[cbuescher]

Build scan:

https://gradle-enterprise.elastic.co/s/xbmzozo4o7tgw
https://gradle-enterprise.elastic.co/s/hymnioyws47co
https://gradle-enterprise.elastic.co/s/ixigz3nn3jsmg

Repro lines:
REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.get/10_basic/Get snapshot info when verbose is false}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.create/10_basic/Create a snapshot}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.create/10_basic/Create a snapshot for missing index}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=cat.snapshots/10_basic/Test cat snapshots output}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.restore/10_basic/Create a snapshot and then restore it}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.status/10_basic/Get snapshot status}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.get/10_basic/Get snapshot info}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.get/10_basic/Get snapshot info contains include_global_state}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.get/10_basic/Get snapshot info when verbose is false}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT"
-Dtests.method="test {p0=snapshot.create/10_basic/Create a snapshot}"
-Dtests.seed=41E70BDA7F09EFAD
-Dtests.security.manager=true
-Dtests.locale=ja-JP
-Dtests.timezone=Europe/Dublin
-Dtests.distribution=default
-Druntime.java=8

Reproduces locally?:

No

Applicable branches:

Seen across all active 7 branches.

Failure history:

Several similar looking failures over the last days, recent three ones linked in the build scans above.

Failure excerpt:

java.lang.AssertionError: expected [2xx] status code but api [snapshot.create] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"null_pointer_exception","reason":null,"stack_trace":"; nested: NullPointerException;\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:149)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:110)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:93)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)\n\tat org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:98)\n\tat 
[...]
Caused by: java.lang.NullPointerException\n\tat org.elasticsearch.cluster.SnapshotsInProgress$Entry.assertShardsConsistent(SnapshotsInProgress.java:159)\n\tat org.elasticsearch.cluster.SnapshotsInProgress$Entry.<init>(SnapshotsInProgress.java:118)\n\tat org.elasticsearch.cluster.SnapshotsInProgress$Entry.<init>(SnapshotsInProgress.java:173)\n\tat org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:277)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219)\n\tat org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\n\tat 

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[original-brownbear]

There's a real bug in the 7.x BwC logic here introduced by the concurrent snapshots work. I'm on it.

[original-brownbear]

Fixed by #60006