Exporter is losing data from time to time

[kennyche]

Getting the same as issue #109
3.7.7 Rabbitmq,
Prometheus 2.10.0,
kbudde/exporter any of the v1.0 RCs.

Alerts were constantly firing whenever data started coming in again.

If i roll back the exporter to 0.29.0 all is fine again.

Tried increasing the timeout to 60/120 seconds with the same results. Can gladly post stdout logs, but no real errors that i can see.

[mindw]

@kennyche got hit by this as well. The recently added label self is likely the root cause when accessing the cluster via a proxy (aka K8s service). A possible workaround is to target a specific instance (preferably the first one).

[kennyche]

@mindw Thanks Gabi, ive reverted back to v0.29.0 for the time being. I'll try pointing the exporter directly to the first instance rather than the svc.

[kbudde]

Hi,
@mindw you are right. The self label is causing issue if the exporter is accessing rabbitmq through a load balancer.

Current workaround is to drop the self label.

- job_name: 'kubernetes_pods'
  kubernetes_sd_configs:
  [...]
  metric_relabel_configs:
  - regex: 'self'
     action: labeldrop

I will try to find a better solution

[kbudde]

Fixed in https://github.com/kbudde/rabbitmq_exporter/releases/tag/v1.0.0-RC19

[kbudde]

See FAQ about required configuration:
https://github.com/kbudde/rabbitmq_exporter/#missing-data-in-graphs