Prometheus TSDB enablement

[tetianakravchenko]

Test Environment Setup

• Creation of Prometheus Test Environment

Datastream : Collector

• Add dimension fields - [Prometheus] Add dimension fields to query and collector datastreams for TSDB support #5974
• Adding metric_type mapping

Verification and validation

• Verification of data in visualisation after enabling TSDB flag in kibana
• Verification of the count of documents (before & after TSDB enablement) in Discover Interface

Datastream : Query

• Add dimension fields - [Prometheus] Add dimension fields to query and collector datastreams for TSDB support #5974
• Adding metric_type mapping

Verification and validation

• Verification of data in visualisation after enabling TSDB flag in kibana
• Verification of the count of documents (before & after TSDB enablement) in Discover Interface

Datastream : Remote Write

⚠️ after migrating to TSDB (labels_id is added) I can see dropped documents - reason: metrics are not correctly grouped by the labels, seems like a bug in metricbeat

related issue - #7533

• Add dimension fields - [Prometheus] [remote_write] Add dimention and metric_type metadata #7565
• Adding metric_type mapping - [Prometheus] [remote_write] Add dimention and metric_type metadata #7565

Verification and validation

• Verification of data in visualisation after enabling TSDB flag in kibana
• Verification of the count of documents (before & after TSDB enablement) in Discover Interface

[tetianakravchenko]

relevant documentation during the TSDB research phase:

some limitation on dimensions fields - https://github.com/elastic/elasticsearch-adrs/blob/master/analytics/tsdb/tsdb-design.md#constraints-on-dimension-fields

TSDB Dimensions proposal for ecs field - elastic/ecs#2172

[tetianakravchenko]

Common

1. For the prometheus integration labels should be used as an dimensions, since labels field is of type object and can exceed the existing limit of 16 fields on _tsid (7 ecs fields are marked as dimension, left only 9), I think it makes sense to introduce the fingerprint processor as mentioned in doc

For example, one of documents that I found has 10 label - together with ecs fields it is already 17 fields - that exceeds the default limit

it is still possible to rewrite the default limit, but since the number of labels is dynamic I think it is better to use fingerprint processor

2. Metric_type for fields with dynamic mapping does not work - [Fleet] TSDB: need metric_type mapping for dynamic templates kibana#155004
   Metric_type is a missing part for all 3 data_streams.

3. Histograms are not supported - after enabling TSDB histogram format has changed:

[tetianakravchenko]

Dimensions

✅ Collector datastream - all works after merging fix PR ✅

1. After enabling TSDB - 1 document is dropped. Fix - [Prometheus] Add up metric to the document with the same set of labels beats#35117
2. Dashboard for Prometheus Server - some gaps, should be fixed by the mentioned above PR - [Prometheus] Add up metric to the document with the same set of labels beats#35117

Query

1. Enabling TSDB - depending on list of queries, can see missing document
   One of 2 use case: using query that returns a single value - in this case no labels will be returned, but labels_id will be still calculated:

• is it fine to add a labels_id on missing labels field?
  • after adding query_name as an extra label labels field shouldn't be missing
• For such cases should be used query name as a dimension, need to extract if from an object - https://github.com/elastic/integrations/blob/main/packages/prometheus/data_stream/query/fields/fields.yml#L9-L14
  • used query_name as an extra label d59d784
• In configuration it is possible to define query with the same name multiple times. For now I would assume that unique names are provided.

2. No dashboards for this data_stream

Remote_write

1. after migrating to TSDB (labels_id is added) I can see dropped documents - reason: metrics are not correctly grouped by the labels, seems like a bug in metricbeat
2. Dashboard is affected due to missing documents

[StephanErb]

@tetianakravchenko are you considering to support use_types and rate_counters here also? Or is use_types implicitly assumed anyway to differentiate between counters, gauges, and histograms?

[lalit-satapathy]

@tetianakravchenko, Can metric_type be now added since elastic/kibana#155004 is fixed?

[lalit-satapathy]

@tetianakravchenko, Can we update this issue/description, where we are wrt. prometheus TSDB migration and testing status?

[tetianakravchenko]

@tetianakravchenko, Can we update this issue/description, where we are wrt. prometheus TSDB migration and testing status?

@lalit-satapathy I've updated description. going to run local tests for collector/query today