RabbitMQ receiver not working

[astingengo]

Component(s)

receiver/rabbitmq

What happened?

Description

Trying to get metrics from RabbitMQ but I see nothing

Steps to Reproduce

Check additional context

Expected Result

Seeing metrics from RabbitMQ

Actual Result

No metrics

Collector version

0.77.0

Environment information

Environment

OS: Ubuntu 22.04
Kubernetes: 1.25.6

OpenTelemetry Collector configuration

receivers:
      rabbitmq:
        endpoint: https://rabbitmq:15671
        username: user
        password: pass
        tls:
          insecure_skip_verify: true
    processors:
      resourcedetection:
          detectors: [system]
      batch:
        send_batch_size: 10000
        timeout: 10s
    exporters:
      otlp/uptrace:
        endpoint: http://my-uptrace:14317
        tls:
          insecure: true
        headers:
          uptrace-dsn: 'http://project1_secret_token@my-uptrace:14317/1'
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        metrics:
          receivers: [rabbitmq]
          processors: [batch, resourcedetection]
          exporters: [otlp/uptrace]

Log output

2023-05-16T09:44:44.227Z    info    service/telemetry.go:113    Setting up own telemetry...                                                                                                                                                                                                                                                                                                                                         │
2023-05-16T09:44:44.228Z    info    service/telemetry.go:136    Serving Prometheus metrics    {"address": ":8888", "level": "Basic"}                                                                                                                                                                                                                                                                                                │
2023-05-16T09:44:44.228Z    debug    exporter@v0.77.0/exporter.go:284    Stable component.    {"kind": "exporter", "data_type": "metrics", "name": "otlp/uptrace"}                                                                                                                                                                                                                                                                  │
2023-05-16T09:44:44.228Z    debug    processor/processor.go:298    Beta component. May change in the future.    {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}                                                                                                                                                                                                                                           │
2023-05-16T09:44:44.228Z    debug    processor/processor.go:298    Stable component.    {"kind": "processor", "name": "batch", "pipeline": "metrics"}                                                                                                                                                                                                                                                                               │
2023-05-16T09:44:44.228Z    debug    receiver@v0.77.0/receiver.go:305    Beta component. May change in the future.    {"kind": "receiver", "name": "rabbitmq", "data_type": "metrics"}                                                                                                                                                                                                                                              │
2023-05-16T09:44:44.229Z    info    service/service.go:141    Starting otelcol-contrib...    {"Version": "0.77.0", "NumCPU": 8}                                                                                                                                                                                                                                                                                                     │
2023-05-16T09:44:44.229Z    info    extensions/extensions.go:41    Starting extensions...                                                                                                                                                                                                                                                                                                                                           │
2023-05-16T09:44:44.229Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] Channel created    {"grpc_log": true}                                                                                                                                                                                                                                                                                                             │
2023-05-16T09:44:44.229Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] original dial target is: "my-uptrace.uptrace.svc:14317"    {"grpc_log": true}                                                                                                                                                                                                                                                                     │
2023-05-16T09:44:44.229Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] parsed dial target is: {Scheme:my-uptrace.uptrace.svc Authority: URL:{Scheme:my-uptrace.uptrace.svc Opaque:14317 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}    {"grpc_log": true}                                                                                                              │
2023-05-16T09:44:44.229Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] fallback to scheme "passthrough"    {"grpc_log": true}                                                                                                                                                                                                                                                                                            │
2023-05-16T09:44:44.230Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/my-uptrace.uptrace.svc:14317 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}    {"grpc_log": true}                                                                                                            │
2023-05-16T09:44:44.230Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] Channel authority set to "my-uptrace.uptrace.svc:14317"    {"grpc_log": true}                                                                                                                                                                                                                                                                     │
2023-05-16T09:44:44.230Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] Resolver state updated: {                                                                                                                                                                                                                                                                                                                         │
  "Addresses": [                                                                                                                                                                                                                                                                                                                                                                                                                    │
    {                                                                                                                                                                                                                                                                                                                                                                                                                               │
      "Addr": "my-uptrace.uptrace.svc:14317",                                                                                                                                                                                                                                                                                                                                                                                       │
      "ServerName": "",                                                                                                                                                                                                                                                                                                                                                                                                             │
      "Attributes": null,                                                                                                                                                                                                                                                                                                                                                                                                           │
      "BalancerAttributes": null,                                                                                                                                                                                                                                                                                                                                                                                                   │
      "Type": 0,                                                                                                                                                                                                                                                                                                                                                                                                                    │
      "Metadata": null                                                                                                                                                                                                                                                                                                                                                                                                              │
    }                                                                                                                                                                                                                                                                                                                                                                                                                               │
  ],                                                                                                                                                                                                                                                                                                                                                                                                                                │
  "ServiceConfig": null,                                                                                                                                                                                                                                                                                                                                                                                                            │
  "Attributes": null                                                                                                                                                                                                                                                                                                                                                                                                                │
} (resolver returned new addresses)    {"grpc_log": true}                                                                                                                                                                                                                                                                                                                                                                           │
2023-05-16T09:44:44.230Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] Channel switches to new LB policy "pick_first"    {"grpc_log": true}                                                                                                                                                                                                                                                                              │
2023-05-16T09:44:44.230Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1 SubChannel #2] Subchannel created    {"grpc_log": true}                                                                                                                                                                                                                                                                                            │
2023-05-16T09:44:44.230Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] Channel Connectivity change to CONNECTING    {"grpc_log": true}                                                                                                                                                                                                                                                                                   │
2023-05-16T09:44:44.230Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING    {"grpc_log": true}                                                                                                                                                                                                                                                                  │
2023-05-16T09:44:44.231Z    info    internal/resourcedetection.go:136    began detecting resource information    {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics"}                                                                                                                                                                                                                                          │
2023-05-16T09:44:44.231Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1 SubChannel #2] Subchannel picks a new address "my-uptrace.uptrace.svc:14317" to connect    {"grpc_log": true}                                                                                                                                                                                                                                      │
2023-05-16T09:44:44.231Z    info    zapgrpc/zapgrpc.go:178    [core] pickfirstBalancer: UpdateSubConnState: 0xc0006dfb00, {CONNECTING <nil>}    {"grpc_log": true}                                                                                                                                                                                                                                                                  │
2023-05-16T09:44:44.231Z    info    internal/resourcedetection.go:150    detected resource information    {"kind": "processor", "name": "resourcedetection", "pipeline": "metrics", "resource": {"host.id":"3c05fe40-34ce-4c93-994a-06f8bbd41d5c","host.name":"otel-collector-rqz2j","os.type":"linux"}}                                                                                                                            │
2023-05-16T09:44:44.231Z    info    service/service.go:158    Everything is ready. Begin running and processing data.                                                                                                                                                                                                                                                                                                               │
2023-05-16T09:44:44.325Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY    {"grpc_log": true}                                                                                                                                                                                                                                                                       │
2023-05-16T09:44:44.325Z    info    zapgrpc/zapgrpc.go:178    [core] pickfirstBalancer: UpdateSubConnState: 0xc0006dfb00, {READY <nil>}    {"grpc_log": true}                                                                                                                                                                                                                                                                       │
2023-05-16T09:44:44.325Z    info    zapgrpc/zapgrpc.go:178    [core] [Channel #1] Channel Connectivity change to READY    {"grpc_log": true}                                                                                                                                                                                                                                                                                        │

Additional context

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-testing
data:
  config.yaml: |-
    receivers:
      rabbitmq:
        endpoint: https://rabbitmq:15671
        username: user
        password: pass
        tls:
          insecure_skip_verify: true
    processors:
      resourcedetection:
          detectors: [system]
      batch:
        send_batch_size: 10000
        timeout: 10s
    exporters:
      otlp/uptrace:
        endpoint: http://my-uptrace:14317
        tls:
          insecure: true
        headers:
          uptrace-dsn: 'http://project1_secret_token@my-uptrace:14317/1'
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        metrics:
          receivers: [rabbitmq]
          processors: [batch, resourcedetection]
          exporters: [otlp/uptrace]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-collector
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.77.0
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
        - mountPath: /etc/otelcol-contrib/config.yaml
          name: data
          subPath: config.yaml
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: data
        configMap:
          name: otel-testing
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  ports:
  - name: metrics
    port: 8888
  selector:
    component: otel-collector

[github-actions]

Pinging code owners:

• receiver/rabbitmq: @djaglowski @cpheps

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Is this the only log line printed, even with logging set to debug?

2023-05-16T09:44:44.227Z    info    service/telemetry.go:113    Setting up own telemetry...   

[astingengo]

The log was not copied correctly so the scroll was horizontal
I was able to copy and paste it vertically now

This is all log

[djaglowski]

Thanks for updating the log. Unfortunately, there's not much to go on here.

My recommendation would be to troubleshoot by isolating components.

• Try swapping in a different receiver (e.g. hostmetrics).
• Try running the rabbitmq in the simplest possible pipeline (e.g. no processors, file or logging exporter).

[astingengo]

I guess I founded the issue but not the solution.

"Cannot create liveness probe.","opentelemetrycollector":"uptrace/rabbitmq","error":"service property in the configuration doesn't contain extensions","stacktrace"

it seems it expects a health check

as if I'll try with empty health check will not work as:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: rabbitmq
  namespace: uptrace
spec:
  image: otel/opentelemetry-collector-contrib:0.77.0
  config: |
    receivers:
      rabbitmq:
        endpoint: http://rabbit:15672
        username: user
        password: pass
        collection_interval: 5s

    processors:

    extensions:
      health_check: {}

    exporters:
      otlp/uptrace:
        endpoint: uptrace:4317
        headers:
          # Copy your project DSN here
          uptrace-dsn: 'http://project3_secret_token@uptrace:14317/2'

    service:
      extensions: [health_check]
      pipelines:
        metrics:
          receivers: [rabbitmq]
          processors: []
          exporters: [otlp/uptrace]

will output

"controllers.OpenTelemetryCollector","msg":"couldn't determine metrics port from configuration, using 8888 default value","opentelemetrycollector":"uptrace/rabbitmq","error":"missing port in address"} 

[djaglowski]

This appears to be related to open-telemetry/opentelemetry-helm-charts#242 and open-telemetry/opentelemetry-helm-charts#697.

From what I can tell, this doesn't appear to be related to rabbitmq. I'll leave the issue open for further discussion but I'd encourage you to engage on the linked issues.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/rabbitmq: @djaglowski @cpheps

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mrofisr]

it's any update for this problem? becase i have some problem when i use RabbitMQ receiver to get data from management API RabbitMQ?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/rabbitmq: @djaglowski @cpheps

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[bryannaegele]

I can't seem to get this working even locally.

rabbitmq:3.11.24-management
otel/opentelemetry-collector-contrib:0.98.0

  otlp:
    protocols:
      grpc:
        endpoint: "otel_collector:55680"
  rabbitmq:
    endpoint: http://rabbitmq:15672
    username: otel
    password: otel
    collection_interval: 10s
processors:
  batch:
    send_batch_size: 1024
    timeout: 5s
exporters:
  debug:
    verbosity: normal
  otlp/grafana:
    endpoint: tempo:4317
    tls:
      insecure: true
extensions:
  zpages:
    endpoint: :55679
service:
  extensions: [zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/grafana]

    metrics:
      receivers: [rabbitmq]
      processors: []
      exporters: [debug]

"users": [
    {
      "name": "otel",
      "password": "otel",
      "tags": "monitoring"
    }
  ]

Connection is made with no errors but no metrics are generated. Adding a hostmetrics receiver verifies everything is set up properly.

2024-04-15 20:34:27 2024-04-16T02:34:27.823Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 0, "metrics": 0, "data points": 0}

Is there any way to debug further?

[bryannaegele]

I found the issue locally. I think these docs are quite misleading. You still have to grant permissions to all the vhosts for the user to be able to access the vhost.

For anyone else looking for an easy way to manage this with new vhosts, this functionality was added in 3.11.11 to automatically grant permissions for a user.