Cache vectors usage stats

[DaveCTurner]

Today VectorsUsageTransportAction is pretty heavyweight since it must
decompress and read the mappings for every index in the cluster. In
particular Metricbeat hits this action every 10s by default, and it runs
on the elected master, which causes nontrivial load in an otherwise
quiet cluster.

This commit introduces a cache for the usage stats, keyed by index,
avoiding recomputing these stats in the common case that the mapping
hasn't changed.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[mayya-sharipova]

@DaveCTurner Thanks very much for your PR, and sorry that this part was a bottleneck for metrics. But I think a better approach should be to completely remove xpack vectors usage stats, as we already report mappings stats of all fields in /_cluster/stats/indices section.

Confirming with @giladgal that it is ok to move from the current:

GET _xpack/usage
...
 "vectors" : {
    "available" : true,
    "enabled" : true,
    "dense_vector_fields_count" : 3,
    "sparse_vector_fields_count" : 0,
    "dense_vector_dims_avg_count" : 3
  }

to cluster stats API:

GET _cluster/stats
...
  {
    "name" : "dense_vector",
    "count" : 3,
    "index_count" : 2
  }

which should be consistent how we report mapping stats for other fields.

@DaveCTurner Sorry for the trouble!

[DaveCTurner]

Not at all, doing no work is always better than doing less work 😁

Can we remove these stats in 7.x tho? It looks like a breaking change to me. It also seems to lose the dense_vector_dims_avg_count field, not sure how important that really is.

[mayya-sharipova]

We have clarified in the past that usage stats updates are not considered breaking, so it is safe to remove those in 7.x

About missing dense_vector_dims_avg_count in the cluster stats, that's what I wanted to confirm with @giladgal. To my mind it is ok to miss this. May be later, we can consider enhancing cluster mapping stats with additional field type specific bits

[giladgal]

About missing dense_vector_dims_avg_count in the cluster stats, that's what I wanted to confirm with @giladgal. To my mind it is ok to miss this. May be later, we can consider enhancing cluster mapping stats with additional field type specific bits

That's fine by me. Thanks.

[mayya-sharipova]

I've created a PR to remove xpack vector usage.

[DaveCTurner]

Great, thanks Mayya. Closing this in favour of #75017