Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate label names during metric conversion to prometheus format #6858

Closed
pomverte opened this issue Dec 27, 2022 · 2 comments
Closed

Duplicate label names during metric conversion to prometheus format #6858

pomverte opened this issue Dec 27, 2022 · 2 comments
Labels
bug Something isn't working

Comments

@pomverte
Copy link

Describe the bug

Otel Collector is configured to scrape its own metrics, convert it to prometheus/openmetrics format

Steps to reproduce

docker-compose.yml

---
version: "2"
services:
  otel-collector:
    container_name: otel
    image: otel/opentelemetry-collector:0.68.0
    command: [ "--config=/etc/otel-collector.yml" ]
    volumes:
      - ./otel-collector.yml:/etc/otel-collector.yml:ro

  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.41.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

otel-collector.yml

---
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          # scrape_interval: 15s
          static_configs:
            - targets: ["localhost:8888"]

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
    send_timestamps: true
    enable_open_metrics: true
    resource_to_telemetry_conversion:
      enabled: true

processors:
  batch:

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheus]

prometheus.yml

---
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: otel-collector
    static_configs:
      - targets: ["otel-collector:8889"]

What did you see instead ?

Error in logs during conversion

2022-12-27T15:11:36.239Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_process_runtime_heap_alloc_bytes: duplicate label names        {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.240Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_process_runtime_total_sys_memory_bytes: duplicate label names  {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.240Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_process_runtime_total_alloc_bytes: duplicate label names       {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_process_uptime: duplicate label names  {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_processor_batch_timeout_trigger_send: duplicate label names    {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_receiver_refused_metric_points: duplicate label names  {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_processor_batch_batch_send_size: duplicate label names {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_process_memory_rss: duplicate label names      {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_exporter_send_failed_metric_points: duplicate label names      {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_exporter_sent_metric_points: duplicate label names     {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_exporter_enqueue_failed_log_records: duplicate label names     {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_exporter_enqueue_failed_spans: duplicate label names   {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.241Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_process_cpu_seconds: duplicate label names     {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.242Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_receiver_accepted_metric_points: duplicate label names {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458
2022-12-27T15:11:36.242Z        error   prometheusexporter@v0.68.0/collector.go:362     failed to convert metric otelcol_exporter_enqueue_failed_metric_points: duplicate label names   {"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.68.0/collector.go:362
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.14.0/prometheus/registry.go:458

What version did you use ?

Version: 0.68.0

Environment

docker / docker-compose

@pomverte pomverte added the bug Something isn't working label Dec 27, 2022
@srikanthccv
Copy link
Member

The collector own metrics add service_instance_id with a random UUID value and service_name build info command. The Prometheus receiver adds the instance host:port as service.instance.id and job name as service.name to the labels map. When they are sanitized there will be duplicate entries for the same keys leading to this error. This is not a new issue with the recent version and it has been an issue for quite some time.

@Aneurysm9
Copy link
Member

This is a duplicate of open-telemetry/opentelemetry-collector-contrib#14900.

@Aneurysm9 Aneurysm9 closed this as not planned Won't fix, can't repro, duplicate, stale Dec 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants