[BUG] testNodesRemovedAfterZoneDecommission_ClusterManagerNotInToBeDecommissionedZone flaky

[dblock]

org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT/testNodesRemovedAfterZoneDecommission_ClusterManagerNotInToBeDecommissionedZone

https://build.ci.opensearch.org/job/gradle-check/6771/
https://build.ci.opensearch.org/job/gradle-check/6731/

[imRishN]

Although I am not able to reproduce locally, but I believe this seem to be happening when the request lands up in a decommissioned node where the leader won't be discovered. Internally, if the node is not specified for the client then it creates a RandomizingClient and sometimes it comes out to be a decommissioned node. We can potentially avoid this by always selecting a node which will always remain in the cluster.

public static Client client(@Nullable String node) {
        if (node != null) {
            return internalCluster().client(node);
        }
        Client client = cluster().client();
        if (frequently()) {
            client = new RandomizingClient(client, random());
        }
        return client;
    }

[imRishN]

Will raise a PR to implement this. This PR would resolve #5221 and #5189 as well

[kartg]

Closing since #5252 has been merged