Description
Background
Over in Elastic APM we're investigating extending our support for application metrics. We already support recording simple, single-value metrics. We're now looking at adding support for histograms, which would be indexed using the histogram field type. Later we may want to add support for ingesting simpler summary metrics, indexed using the proposed aggregate_metric field type.
Problem
Metric names are not known (to us) up front, so we rely on dynamic mapping to give the fields the appropriate field type. Until now we've gotten away without any dynamic templates, as the metrics we support now are all simple single-value ones (counters/gauges).
Because histograms are indexed as an object with a conventional structure ({"values": [...], "counts": [...]}), the field mapping must first be explicitly defined or we'll end up with two distinct numeric fields.
Proposed solution
Extend dynamic templates with a means of matching objects (i.e. "match_mapping_type": "object") only when their fields match some criteria. Something along these lines:
PUT metrics
{
"mappings": {
"dynamic_templates": [
{
"histograms": {
"match_mapping_type": "object",
"match_mapping_fields": {
"values": {"match_mapping_type": "double"},
"counts": {"match_mapping_type": "long"}
},
"mapping": {
"type": "histogram"
}
}
}
]
}
}Ideally there would be a way of specifying that the set of fields is complete, along the lines of "dynamic": "strict" in mappings. For aggregate_metric fields we might
Alternatives
We could impose a path naming scheme, where histogram metrics are given a final field name of "histogram", and then use path_match: "*.histogram" and match_mapping_type: object. This would be a bit of an ugly artefact, particularly as it would end up in the UI through search completions, Metric Explorer dropdowns, etc. Ideally the name would be exactly what the user defined in the first place, which is what we store today.