The type of kafka_producer_connection_count keeps changing between counter and gauge

[tuhao1020]

Component(s)

cmd/otelcontribcol, exporter/prometheus, processor/servicegraph

What happened?

The type of kafka_producer_connection_count keeps changing between counter and gauge

# HELP kafka_producer_connection_count The current number of active connections.
# TYPE kafka_producer_connection_count counter
kafka_producer_connection_count{client_id="producer-1",job="otel-demo-provider",kafka_version="3.1.1",spring_id="kafkaProducerFactory.producer-1"} 1

# HELP kafka_producer_connection_count The current number of active connections.
# TYPE kafka_producer_connection_count gauge
kafka_producer_connection_count{client_id="producer-1",job="otel-demo-provider"} 1

Collector version

0.66.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14317
  otlp/dummy: # Dummy receiver for the metrics pipeline
    protocols:
      grpc:
        endpoint: localhost:65535

processors:
  servicegraph:
    metrics_exporter: prometheus/servicegraph # Exporter to send metrics to
    dimensions: [cluster, namespace] # Additional dimensions (labels) to be added to the metrics extracted from the resource and span attributes
    store: # Configuration for the in-memory store
      ttl: 2s # Value to wait for an edge to be completed
      max_items: 200 # Amount of edges that will be stored in the storeMap      

exporters:
  prometheus/servicegraph:
    endpoint: 0.0.0.0:9091  # to prometheus
  otlp:
    endpoint: http://localhost:4317  # to jaeger
    tls:
      insecure: true 
  logging:
    logLevel: debug    

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [servicegraph]
      exporters: [logging, otlp]
    metrics/servicegraph:
      receivers: [otlp]
      processors: []
      exporters: [prometheus/servicegraph]

Log output

2022-11-24T19:18:02.384+0800	error	prometheusexporter@v0.64.0/log.go:34	error gathering metrics: collected metric kafka_producer_connection_count label:<name:"client_id" value:"producer-1" > label:<name:"job" value:"otel-demo-provider" > label:<name:"kafka_version" value:"3.1.1" > label:<name:"spring_id" value:"kafkaProducerFactory.producer-1" > counter:<value:2 >  should be a Gauge
	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.64.0/log.go:34
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
	github.com/prometheus/client_golang@v1.13.1/prometheus/promhttp/http.go:139
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2109
net/http.(*ServeMux).ServeHTTP
	net/http/server.go:2487
go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
	go.opentelemetry.io/collector@v0.64.0/config/confighttp/compression.go:162
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2109
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.36.4/handler.go:204
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
	go.opentelemetry.io/collector@v0.64.0/config/confighttp/clientinfohandler.go:39
net/http.serverHandler.ServeHTTP
	net/http/server.go:2947
net/http.(*conn).serve
	net/http/server.go:1991

Additional context

No response

[github-actions]

Pinging code owners:

• cmd/otelcontribcol: @codeboten
• exporter/prometheus: @Aneurysm9
• processor/servicegraph: @jpkrohling @mapno

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[tuhao1020]

use javaagent to directly export metrics to prometheus, the kafka_producer_connection_count is shown below:

# TYPE kafka_producer_connection_count gauge
# HELP kafka_producer_connection_count The current number of active connections.
kafka_producer_connection_count{client_id="producer-1"} 2.0 1669287599269

[tuhao1020]

The issue I created in opentelemetry-java-instrumentation before: open-telemetry/opentelemetry-java-instrumentation#7302

[jpkrohling]

@mapno, are those metrics generated by the service graph processor? I don't see this pattern in the readme, but I'm not sure which other place would generate this kind of metric.

[mapno]

Hi @tuhao1020. It seems that those metrics come from a different component than the ones listed in the collector config, is that possible?

[tuhao1020]

@mapno metrics come from javaagent. when I change OTEL_METRICS_EXPORTER to prometheus, the type of kafka_producer_connection_count will not change.

Here is my docker-compose file:

version: "3"
services:
  postgres:
    image: bitnami/postgresql
    ports:
      - "5432:5432"
    environment:
      ALLOW_EMPTY_PASSWORD: "yes"
  redis:
    image: bitnami/redis:7.0.2
    ports:
      - "6379:6379"
    environment:
      ALLOW_EMPTY_PASSWORD: "yes"
  zookeeper:
    image: bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: bitnami/kafka:3.3.1
    ports:
      - "9092:9092"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
  jaeger:
    image: jaegertracing/all-in-one:1.39
    ports:
      - "16686:16686"
      - "4317:4317"
      - "4318:4318"
      - "14250:14250"
      - "9411:9411"
    environment:
      COLLECTOR_ZIPKIN_HOST_PORT: ":9411"
      COLLECTOR_OTLP_ENABLED: "true"
  prometheus:
    image: prom/prometheus:v2.40.2
    ports:
      - "9090:9090"
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.enable-remote-write-receiver
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
  grafana:
    image: grafana/grafana-enterprise:9.2.5
    ports:
      - "3000:3000"
  otel-demo-provider:
    image: otel-demo-provider
    ports:
      - "8080:8080"
      - "9463:9464"
    environment:
      DB_HOST: postgres
      DB_PORT: 5432
      DB_USER: postgres
      DB_PASSWORD: password
      REDIS_HOST: redis
      REDIS_PORT: 6379
      JAVA_OPTS: -javaagent:/agent/opentelemetry-javaagent.jar
      OTEL_PROPAGATORS: b3multi
      OTEL_TRACES_EXPORTER: otlp
      OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://192.168.0.201:14317
      OTEL_METRICS_EXPORTER: otlp
      OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: http://192.168.0.201:14317
      OTEL_TRACES_SAMPLER_ARG: 1
      OTEL_TRACES_SAMPLER: parentbased_always_on
      OTEL_RESOURCE_ATTRIBUTES: service.name=otel-demo-provider
    volumes:
      - ./agent/opentelemetry-javaagent.jar:/agent/opentelemetry-javaagent.jar
    restart: always
    depends_on:
      - postgres
      - redis
  otel-demo-consumer:
    image: otel-demo-consumer
    ports:
      - "8081:8081"
      - "9464:9464"
    environment:
      JAVA_OPTS: -javaagent:/agent/opentelemetry-javaagent.jar
      OTEL_PROPAGATORS: b3multi
      OTEL_TRACES_EXPORTER: otlp
      OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://192.168.0.201:14317
      OTEL_METRICS_EXPORTER: otlp
      OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: http://192.168.0.201:14317
      OTEL_TRACES_SAMPLER_ARG: 1
      OTEL_TRACES_SAMPLER: parentbased_always_on
      OTEL_RESOURCE_ATTRIBUTES: service.name=otel-demo-consumer
    volumes:
      - ./agent/opentelemetry-javaagent.jar:/agent/opentelemetry-javaagent.jar
    restart: always
    depends_on:
      - otel-demo-provider
volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

[tuhao1020]

@jpkrohling It doesn't seem to have anything to do with servicegraph processor

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14318

exporters:
  prometheus:
    endpoint: 0.0.0.0:9093  # to prometheus

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [prometheus]

use above configuration, the problem can be reproduced

[jpkrohling]

If this metric is created by the Java Agent, you might want to report this in another repository. Probably this: https://github.com/open-telemetry/opentelemetry-java-instrumentation

[tuhao1020]

use javaagent to directly export metrics to prometheus, the kafka_producer_connection_count is shown below:

# TYPE kafka_producer_connection_count gauge
# HELP kafka_producer_connection_count The current number of active connections.
kafka_producer_connection_count{client_id="producer-1"} 2.0 1669287599269

@jpkrohling

[jpkrohling]

Thank you for confirming. I'm closing this issue here, please open one with the Java Agent repository.