Prometheus textfile collector for nvidia-smi

This is a zero-dependencies (see below for details) standalone tool collecting metrics using the nvidia-smi (NVIDIA System Management Interface) command and formatting them in a Prometheus compatible style that can be used through the node_exporter's textfile collector.

Below is shown an example for a visualization generated by Grafana using the metrics of two GPUs, showing

• temperature as solid lines
• power draft as dotted lines
• the (intended) fan speed as dashed lines

Zero Dependencies

Or: "Why not using the official Prometheus Python Client?"

The tool is intended to work on minimalistic installations, e.g. we are using it on our Xen / Citrix Hypervisor instances. Those setups come with very basic installs (currently based on CentOS) and the installation of additional tools like pip (which would be required for the Python Client) is not always possible / desirable.

Therefore the only actual dependencies of this collector are already always fulfilled on the relevant systems:

• Python 2.7 - comes with the base OS installation
• nvidia-smi - available as soon as the NVIDIA driver package is installed

Permissions

No root permissions are required to collect the metrics through nvidia-smi, instead having a user that is having write permissions to the textfile collector directory (or actually just a single file therein, to be precise) of node_exporter is sufficient.

One simple solution is to run the script under the same account that is also used for the node_exporter. A possible setup could look like this:

adduser \
    --home-dir /var/lib/node_exporter \
    --comment "Prometheus Node Exporter daemon" \
    --system \
    node_exporter

mkdir -pv /var/lib/node_exporter/textfile_collector
chown -R node_exporter:node_exporter /var/lib/node_exporter

Installation

Assuming you have followed the strategy for the user account outlined above, you can simply clone this repo to /opt/nvsmi-prometheus-textfile/ and use the service file provided in the resources directory to run metrics collection via systemd:

cd /opt/
git clone https://github.com/imcf/nvsmi-prometheus-textfile
cd nvsmi-prometheus-textfile/resources
cp -v nvsmi-prometheus-textfile.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now nvsmi-prometheus-textfile.service

Seriously, Python 2.7? In 2021??

Well, that's what is available on the Citrix Hypervisor default installation that we're running. Let's re-evaluate the situation with the next version.

Metric and Label Naming

See the official Prometheus instructions on writing exporters and metric and label naming for more information.