prometheus_process_collector plugin crash when /var/lib/rabbitmq is noexec #26
Description
As explained in #12 (comment) , the prometheus_process_collector plugin aborts the booting of RabbitMQ when the RABBITMQ_PLUGINS_EXPAND_DIR points to a directory which is mounted with noexec flag (or which is associated with a selinux policy which prevents executing code).
I found the issue trying to run RabbitMQ packaged as an Openshift3 (kubernetes) pod which persists its data to a persistent directory on the host. Since the Centos7 host has SELinux enabled by default, I had to chcon -Rt svirt_sandbox_file_t /data/rabbitmq to make the persistent directory available to the docker container. RabbitMQ booted fine until I modified my Dockerfile to enable the prometheus_process_collector; I then found out that the svirt_sandbox_file_t label on the directory only enables read and writes on the directory, enforcing the equivalent of the "noexec" mount flag. This made the loading of the native code in the plugin fail and aborted the boot.
Simple steps to reproduce:
$ sudo lvcreate -n rabbitmq -L 100M ssd
Logical volume "rabbitmq" created.
$ sudo mkfs.xfs /dev/ssd/rabbitmq
<snip>
$ sudo mkdir -p /tmp/rabbitmq
$ sudo mount -o noexec,nodev,nosuid /dev/ssd/rabbitmq /tmp/rabbitmq
$ sudo chown 999:999 /tmp/rabbitmq # 999 is id of rabbitmq user in the official RabbitMQ image
$ docker run --rm -ti -u 999 -v /tmp/rabbitmq:/var/lib/rabbitmq -e RABBITMQ_BASE=/var/lib/rabbitmq deadtrickster/rabbitmq_prometheus:3.6.9.1
BOOT FAILED
===========
Error description:
{plugin_module_unloadable,"prometheus_process_collector",
{error,on_load_failure}}
Log files (may contain more information):
tty
tty
Stack trace:
[{rabbit_plugins,prepare_dir_plugin,1,
[{file,"src/rabbit_plugins.erl"},{line,241}]},
{rabbit_plugins,'-prepare_plugins/1-lc$^1/1-1-',1,
[{file,"src/rabbit_plugins.erl"},{line,204}]},
{rabbit_plugins,prepare_plugins,1,
[{file,"src/rabbit_plugins.erl"},{line,204}]},
{rabbit,broker_start,0,[{file,"src/rabbit.erl"},{line,293}]},
{rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,417}]},
{init,start_em,1,[]},
{init,do_boot,3,[]}]
=WARNING REPORT==== 18-Apr-2017::18:08:46 ===
The on_load function for module prometheus_process_collector returned {error,
{load_failed,
"Failed to load NIF library: '/var/lib/rabbitmq/mnesia/rabbit@e4519f4d31e4-plugins-expand/prometheus_process_collector-1.0.2/priv/prometheus_process_collector.so: failed to map segment from shared object: Operation not permitted'"}}
=INFO REPORT==== 18-Apr-2017::18:08:46 ===
Error description:
{plugin_module_unloadable,"prometheus_process_collector",
{error,on_load_failure}}
Log files (may contain more information):
tty
tty
Stack trace:
[{rabbit_plugins,prepare_dir_plugin,1,
[{file,"src/rabbit_plugins.erl"},{line,241}]},
{rabbit_plugins,'-prepare_plugins/1-lc$^1/1-1-',1,
[{file,"src/rabbit_plugins.erl"},{line,204}]},
{rabbit_plugins,prepare_plugins,1,
[{file,"src/rabbit_plugins.erl"},{line,204}]},
{rabbit,broker_start,0,[{file,"src/rabbit.erl"},{line,293}]},
{rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,417}]},
{init,start_em,1,[]},
{init,do_boot,3,[]}]
{"init terminating in do_boot",{plugin_module_unloadable,"prometheus_process_collector",{error,on_load_failure}}}
init terminating in do_boot ()
Crash dump is being written to: erl_crash.dump...% $ Workaround: explictly set the RABBITMQ_PLUGINS_EXPAND_DIR environment variable to point to a path where code can be executed, for example RABBITMQ_PLUGINS_EXPAND_DIR=/tmp/rabbitmq-expand-plugins. Note that the parent directory of RABBITMQ_PLUGINS_EXPAND_DIR must be writable by the RabbitMQ user, because RabbitMQ will try to rm -rf ${RABBITMQ_PLUGINS_EXPAND_DIR} on boot.
In other words, run the container like this:
$ docker run --rm -ti -u 999 -v /tmp/rabbitmq:/var/lib/rabbitmq -e RABBITMQ_BASE=/var/lib/rabbitmq -e RABBITMQ_PLUGINS_EXPAND_DIR=/tmp/rabbitmq-plugins-expand deadtrickster/rabbitmq_prometheus:3.6.9.1