Kafka receiver does not render otelcol_kafka_receiver_current_offset correctly

[spadger]

Component(s)

receiver/kafka

What happened?

Description

The Kafka receiver only reports otelcol_kafka_receiver_current_offset metrics for a single partition. e.g. If I have a 10-partition topic, and three collectors, each will render the metric once.

The value reported is correct for the single partition that is being reported on; the rest are just missing

I'm /guessing/ the issues is at kafka_receiver.go->ConsumeClaim(), where a partition tag needs to be introduced, with value=claim.Partition()

Steps to Reproduce

• Create a multi-partition kafka topic
• Produce otlp spans, ensuring all partitions contain records
• Inspect http://:8888/metrics

Expected Result

Each partition has its offset reported, with a partition tag (ideally)

otelcol_kafka_receiver_current_offset{partition=0} 50,001
otelcol_kafka_receiver_current_offset{partition=3} 50,002
otelcol_kafka_receiver_current_offset{partition=6} 50,003
otelcol_kafka_receiver_current_offset{partition=9} 50,004

Actual Result

Only one partition has its offset reported. I omitted some uninteresting tag, but note that name was an empty string

otelcol_kafka_receiver_current_offset{name=""} 50,002

Collector version

0.91.0

Environment information

Environment

OS: linux
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  kafka:
    brokers:
      - some.kafka1:9092
      - some.kafka2:9092
      - some.kafka3:9092
    topic: otlp_spans
    encoding: otlp_proto
    group_id: otel-collector-backend
  
  processors:
    memory_limiter:
      limit_mib: 400
      spike_limit_mib: 60 # 15% of (pod memory - 100mi)
      check_interval: 5s
  exporters:
    otlp:
      endpoint: some-collector.telemetry.svc:4317
      tls:
        insecure: true
  extensions:
    health_check:
      endpoint: 0.0.0.0:13133
  service:
    extensions:
      - health_check
    pipelines:
      traces:
        receivers:
          - kafka
        processors:
          - memory_limiter
        exporters:
          - otlp

Log output

No response

Additional context

No response

[spadger]

To add to the fun, I'd never expect the offset to decrease, but look at the graph. I'm guessing this is because the gauge just represents the last committed offset, so if there are e.g. 4 partitions being ready by this collector, we'll see the value fluctuate between 4 smooth lines