Prometheus receiver spec compliance tracker

[dashpole]

Component(s)

receiver/prometheus

Describe the issue you're reporting

Validate and ensure compliance of the prometheus receiver with https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#prometheus-metric-points-to-otlp

Issues tracking spec-noncompliant behavior:

• [receiver/prometheus] Incomplete normalization with pkg.translator.prometheus.NormalizeName feature gate #23208
• Relax requirement to drop prometheus histograms and summaries without a count opentelemetry-specification#3666
• Prometheus receiver: Implement Instrumentation scope specification #25870

[dashpole]

@open-telemetry/wg-prometheus

[dashpole]

Metric Name

The OpenMetrics MetricFamily Name MUST be added as the Name of the OTLP metric. By default, the name MUST be unaltered, but translation SHOULD provide configuration which, when enabled, removes type (e.g. _total) and unit (e.g. _seconds) suffixes.

Configuration to trim suffixes:

opentelemetry-collector-contrib/receiver/prometheusreceiver/config.go

Line 36 in 40d8176

TrimMetricSuffixes bool `mapstructure:"trim_metric_suffixes"`

Name is unaltered by default, unless configured to trim suffixes:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 317 to 321 in 40d8176

	name := mf.name
	if trimSuffixes {
	name = prometheus.TrimPromSuffixes(name, mf.mtype, mf.metadata.Unit)
	}
	metric.SetName(name)

This is compliant with the specification

[dashpole]

Metric Unit

The OpenMetrics UNIT metadata, if present, MUST be converted to the unit of the OTLP metric. The unit SHOULD be translated from Prometheus conventions to OpenTelemetry conventions by:

• Converting from full words to abbreviations (e.g. "milliseconds" to "ms").
• Special case: Converting "ratio" to "1".
• Converting "foo_per_bar" to "foo/bar".

Unit is converted to the unit of the OTLP:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 323 in 40d8176

metric.SetUnit(mf.metadata.Unit)

The unit is NOT translated from Prometheus conventions to OTel conventions

Tracked by #23208

[dashpole]

Metric Description

The OpenMetrics HELP metadata, if present, MUST be added as the description of the OTLP metric.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 322 in 40d8176

metric.SetDescription(mf.metadata.Help)

This is compliant with the specification

[dashpole]

Metric Type

The OpenMetrics TYPE metadata, if present, MUST be used to determine the OTLP data type, and dictates type-specific conversion rules listed below. Metric families without type metadata follow rules for unknown-typed metrics below.

The type is used to determine the OTel metric type. The individual types will be verified below:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 327 in 40d8176

switch mf.mtype {

This is compliant with the specification

[dashpole]

Counters

A Prometheus Counter MUST be converted to an OTLP Sum with is_monotonic equal to true.

Counters become monotonic sums:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 101 to 103 in 36c2759

	case textparse.MetricTypeCounter:
	// always use float64, as it's the internal data type used in prometheus
	return pmetric.MetricTypeSum, true

This is compliant with the specification

[dashpole]

Gauges

A Prometheus Gauge MUST be converted to an OTLP Gauge.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 105 to 106 in 36c2759

	case textparse.MetricTypeGauge, textparse.MetricTypeUnknown:
	return pmetric.MetricTypeGauge, false

This is compliant with the specification

[dashpole]

Info and StateSet

An OpenMetrics Info metric MUST be converted to an OTLP Non-Monotonic Sum unless it is the target_info metric, which is used to populate resource attributes. An OpenMetrics Info can be thought of as a special-case of the OpenMetrics Gauge which has a value of 1, and whose labels generally stays constant over the life of the process. It is converted to a Non-Monotonic Sum, rather than a Gauge, because the value of 1 is intended to be viewed as a count, which should be summed together when aggregating away labels.

An OpenMetrics StateSet metric MUST be converted to an OTLP Non-Monotonic Sum. An OpenMetrics StateSet can be thought of as a special-case of the OpenMetrics Gauge which has a 0 or 1 value, and has one metric point for every possible state. It is converted to a Non-Monotonic Sum, rather than a Gauge, because the value of 1 is intended to be viewed as a count, which should be summed together when aggregating away labels.

Info and StateSet become non-monotonic sums:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 115 to 116 in 36c2759

	case textparse.MetricTypeInfo, textparse.MetricTypeStateset:
	return pmetric.MetricTypeSum, false

This is compliant with the specification

[dashpole]

Unknown-typed

A Prometheus Unknown MUST be converted to an OTLP Gauge.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 105 to 106 in 36c2759

	case textparse.MetricTypeGauge, textparse.MetricTypeUnknown:
	return pmetric.MetricTypeGauge, false

This is compliant with the specification

[dashpole]

Histograms

A Prometheus Histogram MUST be converted to an OTLP Histogram.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 107 to 108 in 36c2759

	case textparse.MetricTypeHistogram:
	return pmetric.MetricTypeHistogram, true

Multiple Prometheus histogram metrics MUST be merged together into a single OTLP Histogram:

• The le label on the _bucket-suffixed metric is used to identify and order histogram bucket boundaries. Each Prometheus line produces one bucket count on the resulting histogram. Each value for the le label except +Inf produces one bucket boundary.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 292 in 36c2759

default:

We do not validate that the histogram bucket series ends in _bucket. We assume that any series that isn't a sum, count, or created is the _bucket series. I think this is compliant with the specification.

le label is used for bucket boundary:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 76 to 83 in 36c2759

	func getBoundary(metricType pmetric.MetricType, labels labels.Labels) (float64, error) {
	val := ""
	switch metricType {
	case pmetric.MetricTypeHistogram:
	val = labels.Get(model.BucketLabel)
	if val == "" {
	return 0, errEmptyLeLabel
	}

The +inf bucket is ignored:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 108 in 36c2759

for i := 0; i < bucketCount-1; i++ {

• Lines with _count and _sum suffixes are used to determine the histogram's count and sum.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 280 to 298 in 36c2759

	case pmetric.MetricTypeHistogram, pmetric.MetricTypeSummary:
	switch {
	case strings.HasSuffix(metricName, metricsSuffixSum):
	mg.sum = v
	mg.hasSum = true
	case strings.HasSuffix(metricName, metricsSuffixCount):
	// always use the timestamp from count, because is the only required field for histograms and summaries.
	mg.ts = t
	mg.count = v
	mg.hasCount = true
	case strings.HasSuffix(metricName, metricSuffixCreated):
	mg.created = v
	default:
	boundary, err := getBoundary(mf.mtype, ls)
	if err != nil {
	return err
	}
	mg.complexValue = append(mg.complexValue, &dataPoint{value: v, boundary: boundary})
	}

• If _count is not present, the metric MUST be dropped.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 89 to 92 in 36c2759

	func (mg *metricGroup) toDistributionPoint(dest pmetric.HistogramDataPointSlice) {
	if !mg.hasCount {
	return
	}

• If _sum is not present, the histogram's sum MUST be unset.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 138 in 36c2759

if mg.hasSum {

This is compliant with the specification

[dashpole]

Summaries

Prometheus Summary MUST be converted to an OTLP Summary.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 113 to 114 in 36c2759

	case textparse.MetricTypeSummary:
	return pmetric.MetricTypeSummary, true

Multiple Prometheus metrics are merged together into a single OTLP Summary:

• The quantile label on non-suffixed metrics is used to identify quantile points in summary metrics. Each Prometheus line produces one quantile on the resulting summary.

We use the quantile label:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 84 to 88 in 36c2759

	case pmetric.MetricTypeSummary:
	val = labels.Get(model.QuantileLabel)
	if val == "" {
	return 0, errEmptyQuantileLabel
	}

It is used to populate the quantile of the resulting summary:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 190 in 36c2759

for _, p := range mg.complexValue {

We do not validate that the quantile has no suffix. We do ensure it does not have other relevant suffixes (sum, count, created). I believe this still complies with the spec, as non-suffixed metrics are used to identify quantiles.

• Lines with _count and _sum suffixes are used to determine the summary's count and sum.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 280 to 298 in 36c2759

	case pmetric.MetricTypeHistogram, pmetric.MetricTypeSummary:
	switch {
	case strings.HasSuffix(metricName, metricsSuffixSum):
	mg.sum = v
	mg.hasSum = true
	case strings.HasSuffix(metricName, metricsSuffixCount):
	// always use the timestamp from count, because is the only required field for histograms and summaries.
	mg.ts = t
	mg.count = v
	mg.hasCount = true
	case strings.HasSuffix(metricName, metricSuffixCreated):
	mg.created = v
	default:
	boundary, err := getBoundary(mf.mtype, ls)
	if err != nil {
	return err
	}
	mg.complexValue = append(mg.complexValue, &dataPoint{value: v, boundary: boundary})
	}

• If _count is not present, the metric MUST be dropped.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 168 to 174 in 36c2759

	func (mg *metricGroup) toSummaryPoint(dest pmetric.SummaryDataPointSlice) {
	// expecting count to be provided, however, in the following two cases, they can be missed.
	// 1. data is corrupted
	// 2. ignored by startValue evaluation
	if !mg.hasCount {
	return
	}

• If _sum is not present, the summary's sum MUST be set to zero.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 183 in 36c2759

if mg.hasSum {

The sum is unset, which is the same as setting to zero.

This is compliant with the specification

[dashpole]

Dropped Types

The following Prometheus types MUST be dropped:

• OpenMetrics GaugeHistogram

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 117 to 122 in 36c2759

	case textparse.MetricTypeGaugeHistogram:
	fallthrough
	default:
	// including: textparse.MetricTypeGaugeHistogram
	return pmetric.MetricTypeEmpty, false
	}

This is compliant with the specification

[dashpole]

Start Time

Prometheus Cumulative metrics can include the start time using the _created metric as specified in OpenMetrics. When converting Prometheus Counters to OTLP, conversion SHOULD use _created where available.

Histogram + Summary:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 290 to 291 in 36c2759

	case strings.HasSuffix(metricName, metricSuffixCreated):
	mg.created = v

Counter:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 300 to 301 in 36c2759

	if strings.HasSuffix(metricName, metricSuffixCreated) {
	mg.created = v

When no _created metric is available, conversion MUST follow Cumulative streams: handling unknown start time by default.

this logic is complex, and is implemented in metrics_adjuster.go.

Conversion MAY offer configuration, disabled by default, which allows using the process_start_time_seconds metric to provide the start time. Using process_start_time_seconds is only correct when all counters on the target start after the process and are not reset while the process is running.

opentelemetry-collector-contrib/receiver/prometheusreceiver/config.go

Lines 37 to 43 in 36c2759

	// UseStartTimeMetric enables retrieving the start time of all counter metrics
	// from the process_start_time_seconds metric. This is only correct if all counters on that endpoint
	// started after the process start time, and the process is the only actor exporting the metric after
	// the process started. It should not be used in "exporters" which export counters that may have
	// started before the process itself. Use only if you know what you are doing, as this may result
	// in incorrect rate calculations.
	UseStartTimeMetric bool `mapstructure:"use_start_time_metric"`

This is compliant with the specification

[dashpole]

Exemplars

OpenMetrics Exemplars can be attached to Prometheus Histogram bucket metric points and counter metric points. Exemplars on histogram buckets SHOULD be converted to exemplars on OpenTelemetry histograms. Exemplars on counter metric points SHOULD be converted to exemplars on OpenTelemetry sums.

We implement the AppendExemplar function to get exemplars provided on series.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/transaction.go

Line 153 in 36c2759

func (t *transaction) AppendExemplar(_ storage.SeriesRef, l labels.Labels, e exemplar.Exemplar) (storage.SeriesRef, error) {

Exemplars are set on histograms:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 156 in 36c2759

mg.setExemplars(point.Exemplars())

Exemplars are set on counters:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 236 in 36c2759

mg.setExemplars(point.Exemplars())

If present, the timestamp MUST be added to the OpenTelemetry exemplar.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 383 in 36c2759

e.SetTimestamp(timestampFromMs(pe.Ts))

The Trace ID and Span ID SHOULD be retrieved from the trace_id and span_id label keys, respectively. All labels not used for the trace and span ids MUST be added to the OpenTelemetry exemplar as attributes.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 25 to 26 in 36c2759

	traceIDKey = "trace_id"
	spanIDKey = "span_id"

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Lines 386 to 406 in 36c2759

	for _, lb := range pe.Labels {
	switch strings.ToLower(lb.Name) {
	case traceIDKey:
	var tid [16]byte
	err := decodeAndCopyToLowerBytes(tid[:], []byte(lb.Value))
	if err == nil {
	e.SetTraceID(tid)
	} else {
	e.FilteredAttributes().PutStr(lb.Name, lb.Value)
	}
	case spanIDKey:
	var sid [8]byte
	err := decodeAndCopyToLowerBytes(sid[:], []byte(lb.Value))
	if err == nil {
	e.SetSpanID(sid)
	} else {
	e.FilteredAttributes().PutStr(lb.Name, lb.Value)
	}
	default:
	e.FilteredAttributes().PutStr(lb.Name, lb.Value)
	}

This is compliant with the specification

[dashpole]

Instrumentation Scope

Each otel_scope_info metric point present in a batch of metrics SHOULD be dropped from the incoming scrape, and converted to an instrumentation scope. The otel_scope_name and otel_scope_version labels, if present, MUST be converted to the Name and Version of the Instrumentation Scope. Additional labels MUST be added as scope attributes, with keys and values unaltered. Other metrics in the batch which have otel_scope_name and otel_scope_version labels that match an instrumentation scope MUST be placed within the matching instrumentation scope, and MUST remove those labels.

Metrics which are not found to be associated with an instrumentation scope MUST all be placed within an empty instrumentation scope, and MUST not have any labels removed.

Related: open-telemetry/opentelemetry-specification#3660

This is not implemented

[dashpole]

Resource Attributes

When scraping a Prometheus endpoint, resource attributes MUST be added to the scraped metrics to distinguish them from metrics from other Prometheus endpoints. In particular, service.name and service.instance.id, are needed to ensure Prometheus exporters can disambiguate metrics using job and instance labels as described below.

The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes:

OTLP Resource Attribute	Description
service.name	The configured name of the service that the target belongs to
service.instance.id	A unique identifier of the target. By default, it should be the <host>:<port> of the scraped URL

The following attributes SHOULD be associated with scraped metrics as resource
attributes, and MUST NOT be added as metric attributes:

OTLP Resource Attribute	Description
server.address	The <host> portion of the target's URL that was scraped
server.port	The <port> portion of the target's URL that was scraped
url.scheme	http or https

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/prom_to_otlp.go

Lines 35 to 48 in 36c2759

	func CreateResource(job, instance string, serviceDiscoveryLabels labels.Labels) pcommon.Resource {
	host, port, err := net.SplitHostPort(instance)
	if err != nil {
	host = instance
	}
	resource := pcommon.NewResource()
	attrs := resource.Attributes()
	attrs.PutStr(conventions.AttributeServiceName, job)
	if isDiscernibleHost(host) {
	attrs.PutStr(conventions.AttributeNetHostName, host)
	}
	attrs.PutStr(conventions.AttributeServiceInstanceID, instance)
	attrs.PutStr(conventions.AttributeNetHostPort, port)
	attrs.PutStr(conventions.AttributeHTTPScheme, serviceDiscoveryLabels.Get(model.SchemeLabel))

In addition to the attributes above, the target_info metric is used to supply additional resource attributes. If present, target_info MUST be dropped from the batch of metrics, and all labels from the target_info metric MUST be converted to resource attributes attached to all other metrics which are part of the scrape. By default, label keys and values MUST NOT be altered (such as replacing _ with . characters in keys).

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/transaction.go

Lines 271 to 282 in 36c2759

	func (t *transaction) AddTargetInfo(labels labels.Labels) error {
	attrs := t.nodeResource.Attributes()
	for _, lbl := range labels {
	if lbl.Name == model.JobLabel || lbl.Name == model.InstanceLabel || lbl.Name == model.MetricNameLabel {
	continue
	}
	attrs.PutStr(lbl.Name, lbl.Value)
	}
	return nil

This is compliant with the specification

[dashpole]

I realized we actually do drop summaries and histograms without a count. I've updated the summary and histogram comments above with the reference. open-telemetry/opentelemetry-specification#3666 is no longer needed

[dashpole]

The prometheus receiver is now fully compliant with the prometheus specification