Histogram metric support

[axw]

Currently, the metrics API only supports recording single value metrics: counters and gauges. We would like to extend the API to also support ingesting pre-aggregated histograms, storing as the histogram field type.

• Define intake API changes (Extend metrics intake to accept histograms #4884)
  - [ ] Investigate how to implement dynamic field mapping for histograms

For dynamically mapping histograms, we are currently waiting for elastic/elasticsearch#61939.

[axw]

For dynamic field mapping, we could use [match_mapping_type](https://www.elastic.co/guide/en/elasticsearch/reference/master/dynamic-templates.html#match-mapping-type): object.

This might be a bit too lax, as it would assume all objects we index are histograms, and might prevent adding support for aggregate_metric later. An extension of Elasticsearch's match_mapping_type: object to also match on the types of object properties might be helpful here. i.e. also require "values" and "counts" fields, with the appropriate types.

Alternatively we could enforce a specific field naming, e.g. prefix all histograms with "histogram.". That feels like a cop out to me though, forcing implementation details onto the user.

[simitt]

Alternatively we could enforce a specific field naming, e.g. prefix all histograms with "histogram.". That feels like a cop out to me though, forcing implementation details onto the user.

Based on the fixed format for histogram fields, we might also be able to detect histograms when processing the input in the apm-server and then add a specific prefix, so the convention is not on the user but auto-applied. In case users search for their specific metric name this might still be an issue. Maybe nesting all of those metrics under a histogram key might be an option.

[axw]

I wasn't very clear in my comment, but that's what I meant by that bit you quoted. i.e. store everything with "histogram" as the root key. I think this would be the simplest option working with what we've got, but it means that users need to prefix everything with "histogram." in their queries.

Again I wasn't clear, but that's what I meant by "forcing implementation details onto the user". Application developers would record a metric with one name, but then it ends up getting this "histogram." prefix because of implementation details/constraints. Not the end of the world, but I think it would be nice if we could avoid it.

[axw]

Opened an ES feature request: elastic/elasticsearch#51341

[axw]

Ran into this odd behaviour while doing some experimentation yesterday: elastic/elasticsearch#51847

Also I created a branch with support for histogram metrics: https://github.com/axw/apm-server/tree/histogram-metrics. Very hacky, and requires explicitly defining field mappings in the index template.