Skip to content

Improve sharing and diffability of IndexRoutingTable #94933

Open
@DaveCTurner

Description

@DaveCTurner

Today IndexRoutingTable$Builder#build constructs an all-new IndexShardRoutingTable for each shard on every change. High-traffic clusters may have large numbers of shards per index, and today's behaviour yields many duplicate IndexShardRoutingTable instances which together may consume a substantial amount of heap on the elected master.

Moreover IndexRoutingTable is a SimpleDiffable so we must transmit the whole table over the wire on any change, which prevents sharing on the receiving node and therefore yields many duplicate IndexShardRoutingTable instances on the other nodes in the cluster.

We should:

  • Re-use existing IndexShardRoutingTable instances where possible when adjusting the routing table on the elected master
  • Refine the diffability of the IndexRoutingTable so that it avoids sending unchanged IndexShardRoutingTable instances over the wire.

Relates #77466

Metadata

Metadata

Assignees

No one assigned

    Labels

    :Distributed Coordination/AllocationAll issues relating to the decision making around placing a shard (both master logic & on the nodes)>enhancement>tech debtSupportabilityImprove our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better.Team:Distributed (Obsolete)Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions