Prometheus Plugin "rabbitmq_disk_space_available_limit_bytes " parse error

[Sukiyakijango]

hi
used versions: RabbitMQ 3.8.9 Erlang 23.2.2 Prometheus 2.23.0
Prometheus plugin seems to have a conversion problem with "rabbitmq_disk_space_available_limit_bytes", on startuo everything was fine 3GB of the config was converted to bytes... but after 4 days the metric turned from float to string and couldnt be parsed by prometheus anymore making all the metrics of rabbitmq unreachable.

rabbitmq_disk_space_available_limit_bytes "3GB"
# TYPE rabbitmq_erlang_processes_limit gauge
# HELP rabbitmq_erlang_processes_limit Erlang processes limit

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

The limit can be provided as a number of bytes or as an information unit (3GB or similar). The value must have changed from one type to another during those 4 days.

[michaelklishin]

Using the following config file:

disk_free_limit.absolute = 3GB

and the following effective configuration setting (rabbitmq-diagnostics environment):

{disk_free_limit,"3GB"}

the metrics returned by the Prometheus scraping endpoint is

# TYPE rabbitmq_disk_space_available_limit_bytes gauge
# HELP rabbitmq_disk_space_available_limit_bytes Free disk space low watermark in bytes
rabbitmq_disk_space_available_limit_bytes 3000000000

rabbitmqctl eval can be used to query the component involved directly:

rabbitmqctl eval 'rabbit_disk_monitor:get_disk_free_limit().'

which for me returns 3000000000

[michaelklishin]

The metric is collected as a computed value. The limit is computed once when free disk space monitor is started on the node. So the only way this value can change is if the limit is updated somehow.

Node log files will contain traces of node restarts and limit changes via CLI tools. Without this information I don't know how this can be reproduced. We also haven't seen this reported elsewhere.

[Sukiyakijango]

thanks you helped me greatly.. server where not restarted, the only changge was taht the dyntrace oneagent was shutdown
and that seems to be the problem

cli output:

rabbitmqctl eval 'rabbit_disk_monitor:get_disk_free_limit().'
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.

logs:

2021-02-16 10:27:10.146 [error] <0.353.0> ** Generic server rabbit_disk_monitor terminating
** Last message in was update
** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@service_rabbitmq_rmq.2.m3bohqaqvry5krde591vsmcak",3000000000,35065884672,100,10000,#Ref<0.3208902715.1944846337.10844>,false,true,10,120000}
** Reason for termination ==
** {badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,internal_update,1,[{file,"src/rabbit_disk_monitor.erl"},{line,209}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,181}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2021-02-16 10:27:10.146 [error] <0.353.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours crashed with reason: bad argument in call to erlang:list_to_integer("'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'") in rabbit_disk_monitor:parse_free_unix/1 line 260
2021-02-16 10:27:10.146 [error] <0.352.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link("3GB") at <0.353.0> exit with reason bad argument in call to erlang:list_to_integer("'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'") in rabbit_disk_monitor:parse_free_unix/1 line 260 in context child_terminated
2021-02-16 10:27:10.160 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,init,1,[{file,"src/rabbit_disk_monitor.erl"},{line,135}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 10
2021-02-16 10:29:10.191 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 9
2021-02-16 10:31:10.202 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 8
2021-02-16 10:33:10.210 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 7
2021-02-16 10:35:10.219 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 6
2021-02-16 10:37:10.226 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 5
2021-02-16 10:39:10.233 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 4
2021-02-16 10:41:10.239 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 3
2021-02-16 10:43:10.245 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 2
2021-02-16 10:45:10.252 [info] <0.27726.114> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{badarg,[{erlang,list_to_integer,["'/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so'"],[]},{rabbit_disk_monitor,parse_free_unix,1,[{file,"src/rabbit_disk_monitor.erl"},{line,260}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,306}]},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},{line,179}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,689}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,765}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},3221225472}, retries left: 1
2021-02-16 12:48:10.436 [warning] <0.266.0> rabbit_sysmon_handler busy_dist_port <0.633.0> [{name,delegate_management_0},{initial_call,{delegate,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.51173>,unknown}
2021-02-16 16:30:55.820 [info] <0.28673.116> accepting AMQP connection <0.28673.116> (10.120.47.212:46190 -> 10.120.45.173:5672)

[michaelklishin]

So the value was set to something that cannot be parsed by rabbit_disk_monitor: '/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so' (what?). RabbitMQ CLI tools perform validations on the provided values. I don't know how Dynatrace OneAgent does what it does.