Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus Histogram Metrics Label “le” are converted from Double to Int #33402

Open
SkylarLNH1029 opened this issue Jun 5, 2024 · 6 comments

Comments

@SkylarLNH1029
Copy link

Problem:
I recently switched from grafana agent to otel collector to scrape Prometheus metrics.

Inside the Orel collector, I have a Prometheus receiver and a Prometheus remote write exporter.

After this switch, I noticed the values of my Prometheus histogram bucket metric label “le” changed from for example “900.0” to “900”.

My questions:

  1. Does otel directly forward the original Prometheus metrics from receiver to exporter since they don’t require any format change? Or does otel convert Prometheus to otlp when receiving and then convert from otlp to Prometheus when exporting? If that’s the case, would there be any incompatibility in the process?
  2. What caused my labels changing from “900.0” to “900”. Does otel convert double/float to int? Or is there some configuration I need to set?
@SkylarLNH1029 SkylarLNH1029 added the bug Something isn't working label Jun 5, 2024
@evan-bradley evan-bradley transferred this issue from open-telemetry/opentelemetry-collector Jun 5, 2024
@evan-bradley evan-bradley added the receiver/prometheus Prometheus receiver label Jun 5, 2024
Copy link
Contributor

github-actions bot commented Jun 5, 2024

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@dashpole
Copy link
Contributor

dashpole commented Jun 5, 2024

Interesting. The internal representation of histogram boundaries in the collector is a Float64 Slice. So it can't be converting to an int.

This sounds like it might be an issue with the prometheusremotewrite exporter's encoding of the float boundaries. This is what it does today:

boundStr := strconv.FormatFloat(bound, 'f', -1, 64)

The prometheus Go client does this to encode floats in the exposition format: https://github.com/prometheus/common/blob/d726751c69dc327617a6b294f0f1e8d448291d15/expfmt/text_create.go#L480

*bp = strconv.AppendFloat((*bp)[:0], f, 'g', -1, 64)

'g' and 'f' don't seem to make a difference, though: https://go.dev/play/p/KDpgzrHq42O

@dashpole
Copy link
Contributor

dashpole commented Jun 5, 2024

Based on the text format example: https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-format-example, the correct encoding of the float 1.0 is actually just 1 for the le label. I'm inclined to conclude that the current behavior is correct.

Copy link
Contributor

github-actions bot commented Jun 5, 2024

Pinging code owners for pkg/translator/prometheusremotewrite: @Aneurysm9. See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

github-actions bot commented Jun 5, 2024

Pinging code owners for exporter/prometheusremotewrite: @Aneurysm9 @rapphil. See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

github-actions bot commented Aug 5, 2024

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants