Shard allocation awareness and search routing

[jimczi]

Today when shard allocation awareness is activated we try to route searches to nodes that have any awareness attribute in common with the coordinating node. However the routing of these search can have a negative effect on the cluster for different reasons:

• Users can overload a single zone if they don't route the search requests to nodes with different attributes.

• Nodes are picked without checking the stats of the adaptive replica selection so an overloaded node will continue to receive the same amount of traffic even if we have evidence
  (service time) that it behaves poorly compared to other.

For these reasons I'd like to propose three options to enhance the situation:

1. Completely decouple allocation awareness and search routing. We'd apply adaptive replica selection by default (or round robin if it is deactivated) even if nodes have allocation awareness attributes.

2. Apply search routing based on allocation awareness attributes only if the adaptive replica selection is deactivated.

3. Add an option in the search preference to reference the awareness attributes that should be used for routing.

1 might be a bit radical so I have a slight preference for 2 and 3 since users that relies on this feature could restore the current behavior by changing a cluster setting (2) or an option in their search requests (3). The benefit of all these options is that we'd use adaptive replica selection by default in all cases.

[elasticmachine]

Pinging @elastic/es-search

[jpountz]

We discussed this in FixitFriday and agreed that having shard allocation awareness attributes used at search time feels surprising, so we agreed to decouple (option 1). We also agreed that it would be breaking and that we should wait for 8.0 to change the default behavior. However we can add ways for users to opt into this new behavior in 7.x.

There was no appetite to add a replacement for this feature (option 3) due to the fact that it doesn't feel very useful and that we have other solutions to provide good locality that feel superior such as adaptive replica selection and cross-cluster replication.