Do not select dedicated master nodes as gateway nodes for cross cluster search

[abdonpijpelink]

Cross Cluster Search works by first connecting to one of the configured seed nodes to retrieve the cluster state of the remote clusters, after which up to three nodes per remote cluster are selected to be the gateway nodes that will handle the actual search requests.

It is my understanding that any node is a potential gateway node, unless you have configured search.remote.node.attr to explicitly filter out specific nodes.

In line with our general advice to not send any requests to dedicated master nodes, would it be an idea to automatically filter out dedicated master nodes (similar to what our clients do with sniffing)?

[elasticmachine]

Pinging @elastic/es-search-aggs

[javanna]

this makes sense to me, what do you think @s1monw ?

[bleskes]

+1. The default client sniffing does similar and only selects data nodes. Given that all reduce work is done on the local cluster, perhaps that makes sense here too as a default.

[s1monw]

+1

[jasontedor]

+1 here too. We are effectively acting as a client of the remote cluster and should be the same as our other clients.