Aggregations enhancement - better memory usage estimates to avoid circuit breaking

[markharwood]

In a recent discuss forum post a user reported circuit breaking exceptions when using a large number of avg aggregations nested under two layers of terms aggregations.
Internally, the estimated memory cost of handling the request was (badly) estimated to be nearly 3GB in size but the actual size should have been much smaller. In part, this is due to a one-size-fits-all cost which is added for every aggregation which is hard-coded to 5kb in the aggregator base class.

For some aggregations this is often an underestimate (e.g. cardinality agg) while for simple metric aggregations like the avg aggregation it is a large over-estimate (which led to the circuit breaker error in the case mentioned above).

The solution requires

1. A change to AggregatorBase to allow subclasses to override a new "getWeight" method.
2. Changes to metric aggregations like avg to provide a better estimate of their memory cost.

[markharwood]

Follow up - we discussed the particular query that led to this issue and were not convinced it wasn't an abusive query. An estimate of a true final response size was 400mb which led us to consider that the reported over-estimate for the size of the interim working state was in this case not a bad call given the final state produced by the coordinating node would be even larger and (currently) not subjected to a circuit breaker.
It will always be tough to efficiently and accurately estimate sizes (especially when breadth-first and depth-first modes come into play) so perhaps the existing algorithm's over-estimation remains a healthy choice of setting?

[polyfractal]

@elastic/es-search-aggs

[polyfractal]

I think this can be closed, now that we are relying heavily on the real-memory circuit breaker in #46751 and #57042