Fix prometheusexporter droping the OTEL resource labels #2899

krak3n · 2021-04-07T20:30:18Z

Description:
Added support for resource_to_telemetry_conversion to the Promethus Exporter so resource attributes are added as labels to metrics.

Link to tracking Issue: Fixes #2498

Testing:
Added new test cases to ensure resource labels are included in the prometheus metrics output when resource_to_telemetry_conversion is enabled.

Two existing test cases were amended due to their requirement via a type assertion to have access to the underlying prometheusExporter which is no longer the case since this was changed to a *exporterhelper.metricsExporter. To accommodate this a wrapper was added which allowed the type assertion to work and the test cases to pass.

Manual testing was done by building and running the otelcol locally with the following configuration:

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    resource_to_telemetry_conversion:
      enabled: true

processors:
  batch:

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

A service was written to send metrics with a resource to the locally running otelcol and the prometheus metrics endpoint was queried and observed that metrics now had resource labels attached to them, e.g:

# HELP example_bar
# TYPE example_bar counter
example_bar{deployment_environment="development",foo="bar",service_name="foo-srv",service_version="v1.2.3",telemetry_sdk_language="go",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="0.19.0"} 193

Documentation:

The Prometheus Exporter README.md was updated to include documentation about the new resource_to_telemetry_conversion configuration option.

linux-foundation-easycla · 2021-04-07T20:30:22Z

The committers are authorized under a signed CLA.

✅ Chris Reeves (0be56e9, 3286b18, db5077d)

codecov · 2021-04-07T20:36:46Z

Codecov Report

Merging #2899 (1d5a944) into main (13e4566) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #2899   +/-   ##
=======================================
  Coverage   91.63%   91.64%           
=======================================
  Files         312      312           
  Lines       15404    15420   +16     
=======================================
+ Hits        14116    14132   +16     
  Misses        881      881           
  Partials      407      407

Impacted Files	Coverage Δ
exporter/prometheusexporter/config.go	`100.00% <ø> (ø)`
exporter/prometheusexporter/factory.go	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13e4566...1d5a944. Read the comment docs.

…s output

langecode · 2021-04-12T14:19:56Z

exporter/prometheusexporter/README.md

@@ -17,6 +17,8 @@ The following settings can be optionally configured:
 - `send_timestamps` (default = `false`): if true, sends the timestamp of the underlying
  metric sample in the response.
 - `metric_expiration` (default = `5m`): defines how long metrics are exposed without updates
+- `resource_attributes_as_tag` (default = `false`): if set to true will transform all resource


I was looking into this exact same feature with the prometheusremotewriteexporter. Was just wondering - strictly speaking it is "resource attributes as labels"? I guess Prometheus uses the term "label" instead of "tag"?

As @bogdandrutu I'll switch it to a convention thats used over in the contrib repo so things are kept consistent, so this will change too

resource_to_telemetry_conversion: enabled: true

bogdandrutu · 2021-04-12T15:58:44Z

exporter/prometheusexporter/README.md

@@ -17,6 +17,8 @@ The following settings can be optionally configured:
 - `send_timestamps` (default = `false`): if true, sends the timestamp of the underlying
  metric sample in the response.
 - `metric_expiration` (default = `5m`): defines how long metrics are exposed without updates
+- `resource_attributes_as_tag` (default = `false`): if set to true will transform all resource


seems to be tags not tag

bogdandrutu · 2021-04-12T16:01:49Z

exporter/prometheusexporter/config.go

+
+	// ResourceAttributesAsTags, if sMet to true, will use the exporterhelper feature to
+	// transform all resource attributes into metric labels.
+	ResourceAttributesAsTags bool `mapstructure:"resource_attributes_as_tags"`


In contrib where we use this, we have config like:

exporterhelper.ResourceToTelemetrySettings `mapstructure:"resource_to_telemetry_conversion"`

so the config will be:

resource_to_telemetry_conversion: enabled: true

If you don't like that name maybe embed that settings but use the name resource_attributes_as_tags? What do you think?

ah cool, it should be consistent with existing implementations so happy to switch it up to:

resource_to_telemetry_conversion: enabled: true

…om exporterhelper

krak3n · 2021-04-13T15:56:16Z

@bogdandrutu I've pushed the required changes, sorry for the delay 👍

…#2899) Bumps [github.com/hashicorp/vault](https://github.com/hashicorp/vault) from 1.13.0 to 1.13.1. - [Release notes](https://github.com/hashicorp/vault/releases) - [Changelog](https://github.com/hashicorp/vault/blob/main/CHANGELOG.md) - [Commits](hashicorp/vault@v1.13.0...v1.13.1) --- updated-dependencies: - dependency-name: github.com/hashicorp/vault dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

krak3n added 3 commits April 7, 2021 20:36

fix: add resource_attributes_as_tags support prometheusexporter

0be56e9

fix: prometheus exporter test cases

3286b18

fix: yaml indentation

db5077d

krak3n requested a review from a team April 7, 2021 20:30

BenCoughlan15 approved these changes Apr 8, 2021

View reviewed changes

krak3n added 3 commits April 8, 2021 10:04

chore: add test case to ensure resource labels are added to prometheu…

97bbba7

…s output

Merge branch 'main' into fix/prometheus-exporter-resource-labels

fa01a46

fix: missing test case for exporterhelper.NewMetricsExporter error

d7f1076

Aneurysm9 approved these changes Apr 8, 2021

View reviewed changes

langecode reviewed Apr 12, 2021

View reviewed changes

bogdandrutu reviewed Apr 12, 2021

View reviewed changes

refactor: use existing resource_to_telemetry_conversion convention fr…

4179644

…om exporterhelper

dashpole approved these changes Apr 14, 2021

View reviewed changes

Merge branch 'main' into fix/prometheus-exporter-resource-labels

1d5a944

bogdandrutu approved these changes Apr 21, 2021

View reviewed changes

bogdandrutu merged commit 034902c into open-telemetry:main Apr 21, 2021

krak3n deleted the fix/prometheus-exporter-resource-labels branch April 22, 2021 10:26

odeke-em mentioned this pull request May 2, 2021

prometheusremotewriteexporter does not support resource_to_telemetry_conversion #2412

Closed

Aneurysm9 mentioned this pull request May 3, 2021

Track @Aneurysm9 interest and contributions for approver #2820

Closed

skonto mentioned this pull request Oct 13, 2021

queue-proxy not sending some tags on metrics when using opencensus knative/serving#12073

Open

vasireddy99 mentioned this pull request Nov 12, 2021

Enable the resource_to_telemetry_conversion to true in amp config.yaml files aws-observability/aws-otel-collector#739

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix prometheusexporter droping the OTEL resource labels #2899

Fix prometheusexporter droping the OTEL resource labels #2899

krak3n commented Apr 7, 2021 •

edited

Loading

linux-foundation-easycla bot commented Apr 7, 2021 •

edited

Loading

codecov bot commented Apr 7, 2021 •

edited

Loading

langecode Apr 12, 2021

krak3n Apr 12, 2021

bogdandrutu Apr 12, 2021

bogdandrutu Apr 12, 2021

krak3n Apr 12, 2021

krak3n commented Apr 13, 2021

Fix prometheusexporter droping the OTEL resource labels #2899

Fix prometheusexporter droping the OTEL resource labels #2899

Conversation

krak3n commented Apr 7, 2021 • edited Loading

linux-foundation-easycla bot commented Apr 7, 2021 • edited Loading

codecov bot commented Apr 7, 2021 • edited Loading

Codecov Report

langecode Apr 12, 2021

Choose a reason for hiding this comment

krak3n Apr 12, 2021

Choose a reason for hiding this comment

bogdandrutu Apr 12, 2021

Choose a reason for hiding this comment

bogdandrutu Apr 12, 2021

Choose a reason for hiding this comment

krak3n Apr 12, 2021

Choose a reason for hiding this comment

krak3n commented Apr 13, 2021

krak3n commented Apr 7, 2021 •

edited

Loading

linux-foundation-easycla bot commented Apr 7, 2021 •

edited

Loading

codecov bot commented Apr 7, 2021 •

edited

Loading