Enhance rollover API and ILM rollover action to specify max single primary shard size

[dakrone]

Currently when configuring either a manual rollover or rollover via ILM, a user can specify "max_size" as one of the rollover conditions. However, this parameter has an ambiguous name that has led to a lot of confusion about how the size is exactly calculated.

We've also discussed how rollover is used as an aid to generate shards of a certain size, so it would be helpful to have a parameter that was independent of the number of primary shards in an index. (for example, max_size: 50gb currently means you'll have 1 50gb shard by default, but if you change the number of primary shards to 2, then you'll end up with 2 25gb shards).

We should introduce a new parameter that is the maximum size of a single primary shard for the index. For example:

max_single_primary_size: 50gb

(that may not end up being the final name, it's just an example). This parameter means that once a single primary shard (any of them) has hit 50gb, the index should be rolled over.

Once we have this new parameter we can:

• Add support for this parameter to the ILM rollover action
• Switch our shipped templates that use ILM policies to use this parameter
• Tell Fleet and other ILM policy installers about the new parameter to have them switch
• Deprecate the ambiguous max_size parameter and potentially add the existing behavior as a more descriptive name (if we decide we want to keep it)
• Remove max_size from 8.0+ (as a breaking change)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

[joegallo]

Related to #64544

[jethr0null]

cc @elastic/es-ui for awareness

[joegallo]

max_single_primary_size: 50gb

Deprecate the ambiguous max_size parameter and potentially add the existing behavior as a more descriptive name (if we decide we want to keep it)

How about max_shard_size and max_index_size? That is, add the new max_shard_size and rename the current behavior to max_index_size -- while also allowing max_size as is for bwc on 7.x.

[dakrone]

How about max_shard_size and max_index_size?

Those are still ambiguous I think, because it's unclear how things are being added up- is it only primary shards? do replicas count? is it the size of a single primary or all of them? average shard size?