Report stats related to new sizing guidance

[DaveCTurner]

In #86223 we adjusted our guidance about node capacity to be more in line with actual resource usage and constraints given recent changes (#77466). The metrics mentioned in the guidance are a little difficult to understand since they depend on opaque mechanisms like mapping deduplication. We should report the relevant stats directly to avoid any misunderstandings.

In particular, in nodes stats we should report the total number estimated overhead of field mappers on each node, and in cluster stats we should report the total number of fields in the cluster (after deduplication).

Edited (2022-06-09) to change "total number of field mappers" to "total estimated overhead of field mappers", because I expect we'll want to refine our "1kB-per-field" guidance in the future, which we can do if we build this guidance into ES too. Total number of field mappers is also likely still useful.

• Estimated overhead of mappers in node stats Introduce node mappings stats #89807
• Field counts & mapping sizes in cluster stats Report overall mapping size in cluster stats #87556
• Update sizing guidance docs to refer to new stats Redefine section on sizing data nodes #90274

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[DaveCTurner]

#87556 addresses the cluster-wide stats. Adding the node-level stats is a little bit tricky because today everything is done at the shard level, and yet mappers are shared across all shards in an index on each node.

[kingherc]

I briefly discussed with @original-brownbear about this feature, and gave me the following tips on getting started:

• We can introduce a new mappers field under NodeIndicesStats with two values, one for the total field count (note that there's no deduplication for data nodes) and one for the estimated overhead (just multiply the field count by 1KiB).
• In org.elasticsearch.indices.IndicesService#statsByShard function, for each index, use the index Service to get the mapper Service to finally get the count of the fieldMappers. There's a small doubt whether this count contains nested ones, but I will verify.
• Maybe add a test for NodeStats similar to ClusterStatsIT.

[kingherc]

@original-brownbear , I see that somebody can call also _nodes/stats?level=shards for shared-level info. Reading @DaveCTurner 's comment above, I see that mappers are shared across all shards in an index on each node. Thus, I understand that the total number of fields we want to expose should only be part of the node level (under nodes > indices in the final json) and not part of the indices (I mean under nodes > indices > indices in the final json) nor shards level. Feel free to correct me if I am wrong.

[DaveCTurner]

note that there's no deduplication for data nodes

Although there's no deduplication of mappers today I expect this will be subject to improvement in a future version (see #86440). It would be good if the solution on which we land doesn't assume "no deduplication" too fundamentally.

the total number of fields we want to expose should only be part of the node level ... and not part of the indices ... nor shards level

Definitely yes to node-level and no to shard-level, but I think it would make sense to report it at the index level too. That way users can see which indices are contributing most to this memory usage which will help them address it.

[kingherc]

Thanks @DaveCTurner ! I have an implementation underway that exposes the field mapper stats in the _nodes/stats API at both the node & index level, but not the shard level finally. A PR is imminent in the next couple of days. It will look like this:

"field_mappers": {
  "total_count": 58,
  "total_estimated_overhead": "58kb",
  "total_estimated_overhead_in_bytes": 59392
}

And we should be able to extend it with more stats about deduplicated fields. Only minor complication would be to see adding the deduplicated stats only at the node level and not at the index level (since I guess the deduplication would happen across indices if my reasoning is correct). But that should be an implementation complication to handle, rather than an API one.