Enforce higher priority for RepositoriesService ClusterStateApplier

[fcofdez]

This avoids shards allocation failures when the repository instance
comes in the same ClusterState update as the shard allocation.

Backport of #58808

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

[original-brownbear]

@fcofdez looks like the relevant test that this PR adds failed on this PR. Is this a possible failure mode in master as well maybe?

[fcofdez]

It seems like the change introduced in 3b71a31 wasn't ported to 7.x, in particular the failure is triggered by trying to modify DISCOVERY_ZEN_MINIMUM_MASTER_NODES_SETTING in InternalTestCluster.java while the master is still blocked waiting for a third data node to join. does that setting still apply? @ywelsch @original-brownbear

[DaveCTurner]

Indeed #39466 was a master-only change since we must still support Zen1 in 7.x for the purposes of rolling upgrades from 6.x. It seems strange that the cluster hasn't recovered by this point, however; I'll dig deeper.

[fcofdez]

Thanks for the clarification @DaveCTurner. In the test I was forcing the recovery to be held until a third data node joins the cluster, the reason was to avoid race conditions around the registration of the cluster state listener that collects the data to do the assertions. So I think that the behavior is kind of expected, I'm not sure if we have a different way to avoid that possible race condition in 7.x without forcing the recovery to be postponed until an additional node joins.

[DaveCTurner]

I see, sorry for the delay, it's taken me over an hour to shave all the yaks needed to run this darn test (required an IntelliJ upgrade). I understand now.

I think you can avoid these checks with autoManageMasterNodes:

diff --git a/x-pack/plugin/searchable-snapshots/src/test/java/org/elasticsearch/xpack/searchablesnapshots/ClusterStateApplierOrderingTests.java b/x-pack/plugin/searchable-snapshots/src/test/java/org/elasticsearch/xpack/searchablesnapshots/ClusterStateApplierOrderingTests.java
index 8277714a3ec..3a21761fcfb 100644
--- a/x-pack/plugin/searchable-snapshots/src/test/java/org/elasticsearch/xpack/searchablesnapshots/ClusterStateApplierOrderingTests.java
+++ b/x-pack/plugin/searchable-snapshots/src/test/java/org/elasticsearch/xpack/searchablesnapshots/ClusterStateApplierOrderingTests.java
@@ -35,10 +35,13 @@ import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.is;

-@ESIntegTestCase.ClusterScope(scope = TEST, numDataNodes = 2)
+@ESIntegTestCase.ClusterScope(scope = TEST, numDataNodes = 0, autoManageMasterNodes = false)
 public class ClusterStateApplierOrderingTests extends BaseSearchableSnapshotsIntegTestCase {

     public void testRepositoriesServiceClusterStateApplierIsCalledBeforeIndicesClusterStateService() throws Exception {
+        internalCluster().setBootstrapMasterNodeIndex(0);
+        internalCluster().startNodes(2);
+
         final String fsRepoName = "fsrepo";
         final String indexName = "test-index";
         final String restoredIndexName = "restored-index";

[fcofdez]

retest this please

[fcofdez]

retest this please

[fcofdez]

retest this please

[DaveCTurner]

LGTM