Move data tier usage calculation to node level (#100230)

[gmarouli]

Current situation
DataTiersUsageTransportAction executes an internal nodes stats action with all the trimmings:

 client.admin() 
     .cluster() 
     .prepareNodesStats() 
     .all() 
     .setIndices(CommonStatsFlags.ALL) 

This puts a lot of memory pressure to the coordinating node (which in this case is always the elected master) that can cause further instabilities.

Proposed solution

We could trim down the data we need since we only care about docs and store per shard, that would reduce what needs to be kept in memory; however, with the optimisations of the many shards project, we would like to make it even more light weight.

We chose to do this and to push some parts of the calculation to the nodes themselves. Namely, each node sends to the elected master, grouped per preferred tier:

• A set of all indices residing in this node, so it can be used to count them.
• A list of all the primary shard sizes residing in this nodes, so we can calculate statistics on them.
• The count of total shards on the node.
• The count of total docs on the node.
• The total store size.

The elected master then collects the data and aggregates them to one response.

Fixes: #100230

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @gmarouli, I've created a changelog YAML for you.

[gmarouli]

Reopened: #47875

[gmarouli]

@elasticmachine run elasticsearch-ci/part-2

[gmarouli]

@elasticmachine update branch

[gmarouli]

@elasticmachine run elasticsearch-ci/part-2

[gmarouli]

@elasticmachine update branch

[elasticsearchmachine]

Hi @gmarouli, I've updated the changelog YAML for you.

[gmarouli]

I am switching to build kite so I can re-run the failing test quicker.

I haven't been able to find the connection between the failing test and this code and I cannot reproduce it locally. But it appears that it fails consistently only in this PR so there must be something I am missing

[gmarouli]

I cannot figure out why the test is failing. I opened another PR in which I am going to apply the changes step by step and see which one is triggering the test failure.

[gmarouli]

Also I was just able to reproduce it the failure when it runs in the whole suite not as an individual test, I will follow up on this too.

[gmarouli]

Duplicate of #101599