Rabbitmq specific metrics (e.g. rabbitmq_queues ) are missing from metrics response

[choyuri]

On a 3-node clustered rabbitmq version 3.6.11, I enabled prometheus plugins on all three nodes
I see all the beam metrics, but not any of the rabbitmq metrics

Initially tried
prometheus_rabbitmq_exporter v3.6.12.1

then tried
prometheus_rabbitmq_exporter v3.6.9.1

there are no errors in rabbit logs.
prometheus metrics are working.. just missing the rabbitmq metrics

thank you

[deadtrickster]

Hi, that's strange. Could you please show output of rabbitmq-plugins list -v?

[choyuri]

[E*] accept
     Version:     	0.3.3
     Description: 	Accept header(s) for Erlang/Elixir

[e*] amqp_client
     Version:     	3.6.11
     Description: 	RabbitMQ AMQP Client

[e*] cowboy
     Version:     	1.0.4
     Dependencies:	[cowlib]
     Description: 	Small, fast, modular HTTP server.

[e*] cowlib
     Version:     	1.0.2
     Description: 	Support library for manipulating Web protocols.

[E*] prometheus
     Version:     	3.4.0
     Description: 	Prometheus.io client in Erlang

[E*] prometheus_httpd
     Version:     	2.1.4
     Dependencies:	[prometheus,accept]
     Description: 	Prometheus.io inets httpd exporter

[  ] prometheus_process_collector
     Version:     	1.1.0
     Description: 	Prometheus.io process collector
    Collector exports the current state of process metrics including cpu, memory,
    file descriptor usage and native threads count as well as the process start and up times.

[E*] prometheus_rabbitmq_exporter
     Version:     	v3.6.12.1
     Dependencies:	[rabbitmq_management,prometheus]
     Description: 	RabbitMQ Prometheus.io metrics exporter

[  ] rabbitmq_amqp1_0
     Version:     	3.6.11
     Dependencies:	[amqp_client]
     Description: 	AMQP 1.0 support for RabbitMQ

[E*] rabbitmq_auth_backend_http
     Version:     	3.6.7
     Dependencies:	[amqp_client]
     Description: 	RabbitMQ HTTP Authentication Backend

[  ] rabbitmq_auth_backend_ldap
     Version:     	3.6.11
     Description: 	RabbitMQ LDAP Authentication Backend

[  ] rabbitmq_auth_mechanism_ssl
     Version:     	3.6.11
     Description: 	RabbitMQ SSL authentication (SASL EXTERNAL)

[  ] rabbitmq_consistent_hash_exchange
     Version:     	3.6.11
     Description: 	Consistent Hash Exchange Type

[E*] rabbitmq_event_exchange
     Version:     	3.6.11
     Description: 	Event Exchange Type

[  ] rabbitmq_federation
     Version:     	3.6.11
     Dependencies:	[amqp_client]
     Description: 	RabbitMQ Federation

[  ] rabbitmq_federation_management
     Version:     	3.6.11
     Dependencies:	[rabbitmq_management,rabbitmq_federation]
     Description: 	RabbitMQ Federation Management

[  ] rabbitmq_jms_topic_exchange
     Version:     	3.6.11
     Description: 	RabbitMQ JMS topic selector exchange plugin

[E*] rabbitmq_management
     Version:     	3.6.11
     Dependencies:	[amqp_client,cowboy,cowlib,rabbitmq_web_dispatch,
                         rabbitmq_management_agent]
     Description: 	RabbitMQ Management Console

[e*] rabbitmq_management_agent
     Version:     	3.6.11
     Description: 	RabbitMQ Management Agent

[  ] rabbitmq_management_visualiser
     Version:     	3.6.11
     Dependencies:	[rabbitmq_management]
     Description: 	RabbitMQ Visualiser

[  ] rabbitmq_mqtt
     Version:     	3.6.11
     Dependencies:	[amqp_client]
     Description: 	RabbitMQ MQTT Adapter

[  ] rabbitmq_recent_history_exchange
     Version:     	3.6.11
     Description: 	RabbitMQ Recent History Exchange

[  ] rabbitmq_sharding
     Version:     	3.6.11
     Description: 	RabbitMQ Sharding Plugin

[  ] rabbitmq_shovel
     Version:     	3.6.11
     Dependencies:	[amqp_client]
     Description: 	Data Shovel for RabbitMQ

[  ] rabbitmq_shovel_management
     Version:     	3.6.11
     Dependencies:	[rabbitmq_management,rabbitmq_shovel]
     Description: 	Management extension for the Shovel plugin

[e*] rabbitmq_stomp
     Version:     	3.6.11
     Dependencies:	[amqp_client]
     Description: 	RabbitMQ STOMP plugin

[  ] rabbitmq_top
     Version:     	3.6.11
     Dependencies:	[amqp_client,rabbitmq_management]
     Description: 	RabbitMQ Top

[  ] rabbitmq_tracing
     Version:     	3.6.11
     Dependencies:	[rabbitmq_management]
     Description: 	RabbitMQ message logging / tracing

[  ] rabbitmq_trust_store
     Version:     	3.6.11
     Description: 	Client X.509 certificates trust store

[e*] rabbitmq_web_dispatch
     Version:     	3.6.11
     Dependencies:	[cowboy]
     Description: 	RabbitMQ Web Dispatcher

[  ] rabbitmq_web_mqtt
     Version:     	3.6.11
     Dependencies:	[cowboy,rabbitmq_mqtt]
     Description: 	RabbitMQ MQTT-over-WebSockets adapter

[  ] rabbitmq_web_mqtt_examples
     Version:     	3.6.11
     Dependencies:	[rabbitmq_web_dispatch,rabbitmq_web_mqtt]
     Description: 	Rabbit WEB-MQTT - examples

[E*] rabbitmq_web_stomp
     Version:     	3.6.11
     Dependencies:	[cowboy,sockjs,rabbitmq_stomp]
     Description: 	Rabbit WEB-STOMP - WebSockets to Stomp adapter

[  ] rabbitmq_web_stomp_examples
     Version:     	3.6.11
     Dependencies:	[rabbitmq_web_dispatch,rabbitmq_web_stomp]
     Description: 	Rabbit WEB-STOMP - examples

[e*] sockjs
     Version:     	0.3.4
     Description: 	SockJS

[deadtrickster]

Thanks!

More questions:

• have you checked all three nodes? All three don't have rabbit-specific metrics?
• what's the output of this call prometheus_registry:collectors(default).?

[choyuri]

yes checked on all three.
what happened was.. I did a test deploy on node1.. didnt see rabbit metrics
tried node2 same issue
tried node 3 same issue

thanks

node1:
[prometheus_boolean,prometheus_counter,prometheus_gauge,prometheus_histogram,
prometheus_mnesia_collector,prometheus_summary,
prometheus_vm_memory_collector,prometheus_vm_statistics_collector,
prometheus_vm_system_info_collector]

node2:
[prometheus_boolean,prometheus_counter,prometheus_gauge,prometheus_histogram,
prometheus_mnesia_collector,prometheus_summary,
prometheus_vm_memory_collector,prometheus_vm_statistics_collector,
prometheus_vm_system_info_collector]

node3:

[prometheus_boolean,prometheus_counter,prometheus_gauge,prometheus_histogram,
prometheus_mnesia_collector,prometheus_summary,
prometheus_vm_memory_collector,prometheus_vm_statistics_collector,
prometheus_vm_system_info_collector]

[deadtrickster]

So rabbitmq collectors just not registered for some reason...
Could you please now try this call prometheus_registry:register_collector(prometheus_rabbitmq_overview_collector). and look at /metrics again?

[choyuri]

wooh! that worked

[deadtrickster]

ok, so the problem is that rabbitmq_ collectors aren't registered automatically on node startup...

Did you strip debug_info?
What if you restart a node?

Anyway will fix registering soon. Nice catch :-)

[choyuri]

thank you for debugging!

so i just followed the documentation and performed enable plugin steps only
it's a pretty hefty cluster so takes non trivial amount of time for each node to bounce and rejoin :) so was trying to avoid restarting
the workaround is great in the meantime.

thanks again!

[deadtrickster]

Should be fixed here: https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/tag/rabbitmq-3.6.14.1.