Unable to remove automatically added job/instance labels from Prometheus Receiver

[kaushal02]

Description

I'm trying to prevent or remove the automatically added job and instance labels from metrics collected via the Prometheus receiver, but none of the documented approaches work.

My source metrics look like this:

mitigator_connector_rate{datasink="mitigator_api_edge",instance_id="my-local",pipeline="mitigator_unified_api_edge",pool_id="default"} 10.0

But after collection through OTEL, they get additional labels:

mitigator_connector_rate{datasink="mitigator_api_edge",instance="localhost:9122",instance_id="my-local",job="defender",pipeline="mitigator_unified_api_edge",pool_id="default"} 10.0

Configuration

I've tried multiple approaches to remove these labels, none of which worked:

1. Using metric_relabel_configs:

scrape_configs:
  - job_name: 'defender'
    metric_relabel_configs:
      - regex: "job|instance"
        action: labeldrop

2. Using honor_labels:

scrape_configs:
  - job_name: 'defender'
    honor_labels: true

3. Using transform processor:

processors:
  transform:
    metric_statements:
      - context: datapoint
        statements:
          - drop_labels(["job", "instance"])

4. Using resource processor:

processors:
  resource:
    attributes:
      - key: job
        action: delete
      - key: instance
        action: delete

5. Using attributes processor:

processors:
  attributes:
    actions:
      - key: job
        action: delete
      - key: instance
        action: delete

Expected Behavior

The metrics should be collected and exported without the automatically added job and instance labels, preserving only the original labels from the source metrics.

Actual Behavior

The job and instance labels persist regardless of configuration attempts to remove them.

Questions

1. Is there a supported way to prevent these labels from being added in the first place?
2. If not, what is the correct approach to remove them?
3. Could the documentation be improved to clarify how to handle this common use case?

Environment

• OpenTelemetry Collector Contrib version: 0.112.0
• Platform: Linux

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins: ["*"]
          allowed_headers: ["*"]
  prometheus:
    config:
      scrape_configs:
        - job_name: 'defender'
          scrape_interval: 10s
          static_configs:
            - targets: ['localhost:9122']

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  prometheus:
    endpoint: "0.0.0.0:9123"

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheus]

Log output

No response

Additional context

These labels are not needed in our use case and add unnecessary complexity to our metrics. While they don't affect functionality, they make metrics harder to work with and increase cardinality unnecessarily.

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bacherfl]

Hi @kaushal02!

Just looked into this - I believe the reason those labels can not be removed is because they are added by the prometheusexporter, based on the service.name and service.instance.id attributes of the related resource - see

opentelemetry-collector-contrib/exporter/prometheusexporter/collector.go

Line 340 in 5258d98

// Map service.name + service.namespace to job

This is specific to the prometheus exporter, as the job/instance labels are used by the prometheus components to associate a prometheus metric to a OpenTelemetry Resource (i.e. job <-> service.name and instance <-> service.instance.id) - You can verify this by, e.g. adding the debug exporter to the pipeline, and there those two labels are missing - this also explains why e.g. the transform processor did not remove those labels, as they were re-added after the processor.
I'm not sure however, if it should be possible to remove those two particular labels in the prometheus exporter, given their purpose in establishing the connection to the resource for a given metric. Here I will have to refer to the code owners of the prometheus exporter @dashpole @Aneurysm9 as they have more insights here.

[github-actions]

Pinging code owners for exporter/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

@bacherfl is correct. If you want to remove those labels, you would need to remove the service.name and service.instance.id resource attributes (e.g. using the resource processor). However, removing those attributes may make it difficult to use your metrics, and may result in collisions (two timeseries with the exact same list of labels).