[RW Separation] Restrict Search Replica Allocation to Search-Dedicated Nodes

[vinaykpud]

Describe the bug

When an index is created or when search replicas are added for an index, search replicas should be allocated only to nodes that have the search role assigned. If there are no such nodes available, the search replicas should remain in an unassigned state. If a node with the search role becomes available later, the search replicas should be assigned accordingly.

Related component

Search:Performance

To Reproduce

1. Create a cluster with 3 nodes.
2. Create an index with the following configuration:
   • 1 Primary shard (1P)
   • 1 Replica shard (1R)
   • 1 Search Replica shard (1SR)
3. Observe how the shards are allocated across the cluster.

• Currently, search replicas may get assigned to nodes without the search role, leading to incorrect shard placement.

Expected behavior

• The primary and replica shards should be assigned to available nodes.
• The search replica should remain in an unassigned state if no node with the search role exists.
• If a node with the search role is added to the cluster, the search replica should be assigned to it.

Assigning a search node role for a node:
Ref: #15445

curl -X PUT "http://localhost:9200/_cluster/settings" \
-H "Content-Type: application/json" \
-d '{
  "persistent": {
    "cluster.routing.allocation.search.replica.dedicated.include._name": "node-1"
  }
}'

Additional Details

No response

[mch2]

@vinaykpud Search replicas rely on the allocation filter for hardware designation, not the "Search node role", are you seeing the bug when applying the filter?

[gbbafna]

@mch2 , @vinaykpud : Adding allocation filter explicitly adds too much overhead . Is there a way to simplify the experience ? Also how would allocation/balancing work out for search replicas ?

[mch2]

Adding allocation filter explicitly adds too much overhead

@gbbafna Can you elaborate here? Are you referring to potential overhead on match with DiscoveryNodeFilter? If thats significant perhaps we limit to a concrete node attr and look up on the DiscoveryNode like isRemoteStoreNode and yank the filter setting.

The idea was to use SearchReplicaAllocationDecider a decider we introduced a while back that operates similarly to FilterAllocationDecider. This would work with any node attribute and not be reliant on a node role, but I'm ok with narrowing this a bit to a specific node attr...

In terms of general allocation/balancing for SRs, we would apply existing deciders, zonal awareness & replica auto expansion etc only within the nodes marked with the attribute. Non SRs would apply the same within their set of nodes.

[vinaykpud]

@vinaykpud Search replicas rely on the allocation filter for hardware designation, not the "Search node role", are you seeing the bug when applying the filter?

This is to prevent search replica allocation in the absence of the include filter.

[gbbafna]

@gbbafna Can you elaborate here? Are you referring to potential overhead on match with DiscoveryNodeFilter? If thats significant perhaps we limit to a concrete node attr and look up on the DiscoveryNode like isRemoteStoreNode and yank the filter setting.

I was referring to overhead on cluster operator to apply this settings . They will need to add either an attribute to all such nodes by defining this in their yml file and use this attribute in their cluster settings. The other option would be to specify nodes , which will be inconvenient .