Skip to content

Computing IndicesQueryCache stats is O(N²) in shard count #97222

Open
@DaveCTurner

Description

@DaveCTurner

We loop over all shards in org.elasticsearch.indices.IndicesService#statsByShard calling indexShardStats for each one, but then further down the stack in org.elasticsearch.indices.IndicesQueryCache#getStats we loop over all the shards again in order to compute the portion of the shared RAM usage to attribute to the current shard. These days a node can hold many thousands of shards, so this duplicated work consumes quite some resources.

We have the same loop in org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction#nodeOperation.

We have effectively the same loop in org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction#shardOperation, but this one is trickier because the outer loop is within TransportBroadcastByNodeAction which doesn't currently have a facility for sharing any context between invocations on different shards.

Relates #77466.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions