grafana-stack

A Docker Compose monitoring stack for Docker hosts and containers, built on Prometheus, Grafana, cAdvisor, Node Exporter, Pushgateway, InfluxDB and alerting with Alertmanager. One docker compose up starts the whole stack, with a Prometheus data source and four Grafana dashboards already provisioned.

This repository is based on stefanprodan/dockprom (see Credits).

Architecture

All services run on a single bridge network, monitor-net, and reach each other by container name.

flowchart LR
    host[(Docker host<br/>/proc, /sys, /)] --> nodeexporter[Node Exporter :9100]
    docker[(Docker engine<br/>/var/lib/docker)] --> cadvisor[cAdvisor :8080]
    jobs[Batch jobs / scripts] -->|push| pushgateway[Pushgateway :9091]
    nodeexporter -->|scrape 5s| prometheus[Prometheus]
    cadvisor -->|scrape 5s| prometheus
    pushgateway -->|scrape 10s| prometheus
    prometheus -->|alerts| alertmanager[Alertmanager :9093]
    alertmanager -->|notify| slack[Slack]
    grafana[Grafana :3000] -->|query| prometheus
    influxdb[InfluxDB :8086]

InfluxDB runs alongside the Prometheus stack, but nothing in this repository writes to it, and it is not configured as a Grafana data source. Add it yourself if you want to use it (for example, as a Grafana data source or as a target for your own collectors).

Services

Service	Image	Host port → container port	Purpose
`prometheus`	`prom/prometheus:v2.45.0`	`9090` → `9000` (see Known issues)	Metrics database and alert rule evaluation. Retention is `200h`; `--web.enable-lifecycle` is on.
`alertmanager`	`prom/alertmanager:v0.25.0`	`9093` → `9093`	Alert routing, grouping and silencing.
`grafana`	`grafana/grafana:10.0.1`	`3000` → `3000`	Dashboards (visualize metrics).
`influxdb`	`influxdb:1.8.1`	`8086` → `8086`	High-speed time series database. Creates the database `influx` on first start.
`pushgateway`	`prom/pushgateway:v1.6.0`	`9091` → `9091`	Push acceptor for ephemeral and batch jobs.
`nodeexporter`	`prom/node-exporter:v1.6.0`	`9100` → `9100`	Host metrics collector (CPU, memory, disk, network).
`cadvisor`	`gcr.io/cadvisor/cadvisor:v0.47.1`	`8080` → `8080`	Container metrics collector. Runs `privileged`.

Every service carries the label org.label-schema.group: "monitoring". The dashboards use this label to tell the stack's own containers apart from your workloads.

Persistent data is stored in bind-mounted directories next to the compose file:

Path on host	Used by
`./prometheus_data`	Prometheus TSDB
`./grafana/data`	Grafana database (users, settings, dashboards saved in the UI)
`./influxdb/data`	InfluxDB data

Requirements

A Linux Docker host (cAdvisor needs /dev/kmsg, /sys and /var/lib/docker).
Docker Engine 19.03 or later, and Docker Compose v2 (docker compose) or docker-compose 1.27 or later. The compose file declares version: '3.9', which Compose v2 ignores.
Free host ports: 3000, 8080, 8086, 9090, 9091, 9093, 9100.

Install

Clone this repository on your Docker host, cd into the grafana-stack directory and run compose up:

git clone https://github.com/t0mer/grafana-stack
cd grafana-stack

ADMIN_USER='admin' ADMIN_PASSWORD='<choose-a-password>' docker compose up -d

(With the older standalone binary, use docker-compose instead of docker compose.)

Endpoints:

Prometheus (metrics database) http://<host-ip>:9090 (see Known issues)
Prometheus Pushgateway (push acceptor for ephemeral and batch jobs) http://<host-ip>:9091
Alertmanager (alerts management) http://<host-ip>:9093 (see Known issues)
Grafana (visualize metrics) http://<host-ip>:3000
InfluxDB (high-speed read and write time series database) http://<host-ip>:8086
cAdvisor (container metrics collector) http://<host-ip>:8080
Node Exporter (host metrics collector) http://<host-ip>:9100/metrics

Configuration

Environment variables

These variables are read by docker-compose.yml. Set them in your shell or in a .env file next to the compose file.

Variable	Default	Used by	Description
`ADMIN_USER`	`admin`	Grafana (`GF_SECURITY_ADMIN_USER`)	Grafana admin user name.
`ADMIN_PASSWORD`	`admin`	Grafana (`GF_SECURITY_ADMIN_PASSWORD`)	Grafana admin password. Change it.
`INFLUX_ADMIN_USER`	`admin`	InfluxDB (`INFLUXDB_ADMIN_USER`)	InfluxDB admin user created on first start.
`INFLUX_ADMIN_PASSWORD`	`admin`	InfluxDB (`INFLUXDB_ADMIN_PASSWORD`)	InfluxDB admin password. Change it.

Example .env:

ADMIN_USER=admin
ADMIN_PASSWORD=<choose-a-password>
INFLUX_ADMIN_USER=admin
INFLUX_ADMIN_PASSWORD=<choose-a-password>

Fixed settings in the compose file:

Grafana: GF_USERS_ALLOW_SIGN_UP=false (self sign-up disabled).
InfluxDB: INFLUXDB_DB=influx (database created on first start).

Configuration files

File	What it configures
`prometheus/prometheus.yml`	Global scrape/evaluation interval (`15s`), external label `monitor: 'docker-host-alpha'`, scrape jobs, Alertmanager target.
`prometheus/alert.rules`	Prometheus alert rules (see Define alerts).
`alertmanager/config.yml`	Alertmanager route and receivers (a Slack receiver template).
`grafana/provisioning/datasources/datasource.yml`	The default Prometheus data source.
`grafana/provisioning/dashboards/`	Dashboard provider (`dashboard.yml`) and the dashboard JSON files.
`influxdb/influxdb.conf`	InfluxDB meta, data and WAL directories.
`config`	Optional Grafana env file (see Setup Grafana). Not referenced by `docker-compose.yml` by default.

Prometheus scrape jobs

Job	Target	Interval
`nodeexporter`	`nodeexporter:9100`	5s
`cadvisor`	`cadvisor:8080`	5s
`prometheus`	`localhost:9090`	10s
`pushgateway`	`pushgateway:9091` (`honor_labels: true`)	10s

Prometheus sends alerts to alertmanager:9093. prometheus.yml also contains commented-out example jobs for nginx (nginxexporter:9113) and aspnetcore.

Setup Grafana

Navigate to http://<host-ip>:3000 and log in with the user and password set by ADMIN_USER and ADMIN_PASSWORD (both default to admin). You can change the credentials in the compose file or by supplying the ADMIN_USER and ADMIN_PASSWORD environment variables on compose up.

You can also use an env file. The repository ships one, named config, with this format:

GF_SECURITY_ADMIN_USER=admin
GF_SECURITY_ADMIN_PASSWORD=changeme
GF_USERS_ALLOW_SIGN_UP=false

To use it, add it to the grafana service:

grafana:
  image: grafana/grafana:10.0.1
  env_file:
    - config

Values in environment: take precedence over env_file:, so also remove the matching GF_* entries from the service's environment: list, otherwise the file has no effect.

The admin credentials are only applied when Grafana creates its database on first start. If you want to change the password later, either change it in the Grafana UI, or reset it from the command line:

docker exec -it grafana grafana-cli admin reset-admin-password '<new-password>'

Alternatively, remove the Grafana data directory (this deletes all Grafana state) so that the environment variables are applied again. It is mounted by this entry:

- ./grafana/data:/var/lib/grafana

Grafana is preconfigured with dashboards and Prometheus as the default data source:

Name: Prometheus
Type: Prometheus
URL: http://prometheus:9090

Dashboards

The dashboards are provisioned from grafana/provisioning/dashboards/ into the General folder. They are editable, and changes can be saved from the UI (allowUiUpdates: true).

Dashboard	File	Data needed
Docker Host	`docker_host.json`	Node Exporter, cAdvisor
Docker Containers	`docker_containers.json`	cAdvisor, Node Exporter
Monitor Services	`monitor_services.json`	cAdvisor, Node Exporter, Prometheus self-metrics
Nginx	`nginx_container.json`	An Nginx exporter scrape job (not enabled by default, see the commented `nginx` job in `prometheus.yml`), cAdvisor and Node Exporter

Docker Host Dashboard

The Docker Host Dashboard shows key metrics for monitoring the resource usage of your server:

Server uptime, CPU idle percent, number of CPU cores, available memory, swap and storage
System load average graph, running and blocked by IO processes graph, interrupts graph
CPU usage graph by mode (guest, idle, iowait, irq, nice, softirq, steal, system, user)
Memory usage graph by distribution (used, free, buffers, cached)
IO usage graph (read Bps, write Bps and IO time)
Network usage graph by device (inbound Bps, outbound Bps)
Swap usage and activity graphs

For storage and particularly the Free Storage panel, you have to specify the fstype in the Grafana query. You can find it in grafana/provisioning/dashboards/docker_host.json, at line 480:

"expr": "sum(node_filesystem_free_bytes{fstype=\"aufs\"})",

The dashboard ships with aufs. If your host uses another filesystem, for example BTRFS, change aufs to btrfs.

You can find the right value for your system in Prometheus by running this query:

node_filesystem_free_bytes

Docker Containers Dashboard

The Docker Containers Dashboard shows key metrics for monitoring running containers:

Total containers CPU load, memory and storage usage
Running containers graph, system load graph, IO usage graph
Container CPU usage graph
Container memory usage graph
Container cached memory usage graph
Container network inbound usage graph
Container network outbound usage graph

Note that this dashboard doesn't show the containers that are part of the monitoring stack (those with the org.label-schema.group="monitoring" label).

For storage and particularly the Storage Load panel, you have to specify the fstype in the Grafana query. You can find it in grafana/provisioning/dashboards/docker_containers.json, at line 406:

"expr": "(node_filesystem_size_bytes{fstype=\"aufs\"} - node_filesystem_free_bytes{fstype=\"aufs\"}) / node_filesystem_size_bytes{fstype=\"aufs\"}  * 100",

As above, change aufs to the filesystem your host uses (for example, btrfs).

You can find the right value for your system in Prometheus by running these queries:

node_filesystem_size_bytes
node_filesystem_free_bytes

Monitor Services Dashboard

The Monitor Services Dashboard shows key metrics for monitoring the containers that make up the monitoring stack:

Prometheus uptime, monitoring stack total memory usage, Prometheus in-memory chunks and series
Container CPU usage graph
Container memory usage graph
Prometheus alerts graph (firing alerts by name)
Prometheus process graphs: file descriptors, memory, allocations
Prometheus TSDB graphs: head series, active appenders, samples appended, head chunks created and removed, head time range, head GC time, blocks loaded, reloads, problems, WAL latencies, compactions, compaction time, retention cutoffs
Prometheus target scrapes and scrape duration graphs
Prometheus HTTP requests, request latency and time spent in HTTP requests
Query engine timings and rule group evaluation graphs

Nginx Dashboard

Shows Nginx requests per second, connections by state, connection rate by stage and the CPU usage of the container named nginx. The nginx_connections_* metrics it queries come from discordianfish/nginx_exporter; nginxinc/nginx-prometheus-exporter exposes different metric names and won't populate it. The dashboard stays empty until you run that exporter and enable a matching scrape job.

Screenshots

The images in screenshots/ come from the author's Grafana instance. Only two of them show dashboards provisioned by this repository:

Screenshot	Dashboard	Provisioned here?
containers.png	Docker Containers	Yes
monitored services.png	Monitor Services	Yes
node exporter.png	A Node Exporter dashboard (not the provisioned Docker Host dashboard)	No
adguard.png	AdGuard Home	No
speedtest.png	Speedtest	No
uptime kuma.png	Uptime Kuma	No

The dashboards that are not provisioned need their own exporters and dashboard JSON, which are not part of this repository.

Define alerts

Three alert groups have been set up within the alert.rules configuration file:

Monitoring services alerts targets
Docker Host alerts host
Docker Containers alerts containers

Group	Alert	Condition	For	Severity
`targets`	`monitor_service_down`	`up == 0` (any scrape target down)	30s	critical
`host`	`high_cpu_load`	`node_load1 > 1.5`	30s	warning
`host`	`high_memory_load`	Host memory usage > 85%	30s	warning
`host`	`high_storage_load`	Storage usage > 85% on `fstype="aufs"`	30s	warning
`containers`	`jenkins_down`	No metrics for a container named `jenkins`	30s	critical
`containers`	`jenkins_high_cpu`	`jenkins` container uses > 10% of total CPU	30s	warning
`containers`	`jenkins_high_memory`	`jenkins` container uses > 1.2 GB of RAM	30s	warning

The containers group is an example written for a container named jenkins. If you don't run one, jenkins_down fires permanently, so adapt the container name or remove these rules.

You can modify the alert rules and reload them without restarting Prometheus. The lifecycle API is enabled (--web.enable-lifecycle), so you can make an HTTP POST call to Prometheus:

curl -X POST http://<host-ip>:9090/-/reload

If the Prometheus port is not reachable from the host (see Known issues), send a SIGHUP to the container instead:

docker kill --signal=HUP prometheus

Monitoring services alerts

Trigger an alert if any of the monitoring targets (node-exporter, cAdvisor, Prometheus, Pushgateway) is down for more than 30 seconds:

- alert: monitor_service_down
  expr: up == 0
  for: 30s
  labels:
    severity: critical
  annotations:
    summary: "Monitor service non-operational"
    description: "Service {{ $labels.instance }} is down."

Docker Host alerts

Trigger an alert if the Docker host CPU is under high load for more than 30 seconds:

- alert: high_cpu_load
  expr: node_load1 > 1.5
  for: 30s
  labels:
    severity: warning
  annotations:
    summary: "Server under high load"
    description: "Docker host is under high load, the avg load 1m is at {{ $value}}. Reported by instance {{ $labels.instance }} of job {{ $labels.job }}."

Modify the load threshold based on your CPU cores.

Trigger an alert if the Docker host memory is almost full:

- alert: high_memory_load
  expr: (sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) ) / sum(node_memory_MemTotal_bytes) * 100 > 85
  for: 30s
  labels:
    severity: warning
  annotations:
    summary: "Server memory is almost full"
    description: "Docker host memory usage is {{ humanize $value}}%. Reported by instance {{ $labels.instance }} of job {{ $labels.job }}."

Trigger an alert if the Docker host storage is almost full (change aufs to your filesystem type, as for the dashboards):

- alert: high_storage_load
  expr: (node_filesystem_size_bytes{fstype="aufs"} - node_filesystem_free_bytes{fstype="aufs"}) / node_filesystem_size_bytes{fstype="aufs"}  * 100 > 85
  for: 30s
  labels:
    severity: warning
  annotations:
    summary: "Server storage is almost full"
    description: "Docker host storage usage is {{ humanize $value}}%. Reported by instance {{ $labels.instance }} of job {{ $labels.job }}."

Docker Containers alerts

Trigger an alert if a container is down for more than 30 seconds:

- alert: jenkins_down
  expr: absent(container_memory_usage_bytes{name="jenkins"})
  for: 30s
  labels:
    severity: critical
  annotations:
    summary: "Jenkins down"
    description: "Jenkins container is down for more than 30 seconds."

Trigger an alert if a container is using more than 10% of total CPU cores for more than 30 seconds:

- alert: jenkins_high_cpu
  expr: sum(rate(container_cpu_usage_seconds_total{name="jenkins"}[1m])) / count(node_cpu_seconds_total{mode="system"}) * 100 > 10
  for: 30s
  labels:
    severity: warning
  annotations:
    summary: "Jenkins high CPU usage"
    description: "Jenkins CPU usage is {{ humanize $value}}%."

Trigger an alert if a container is using more than 1.2 GB of RAM for more than 30 seconds:

- alert: jenkins_high_memory
  expr: sum(container_memory_usage_bytes{name="jenkins"}) > 1200000000
  for: 30s
  labels:
    severity: warning
  annotations:
    summary: "Jenkins high memory usage"
    description: "Jenkins memory consumption is at {{ humanize $value}}."

Setup alerting

The Alertmanager service is responsible for handling alerts sent by the Prometheus server. Alertmanager can send notifications via email, Pushover, Slack, Telegram, Microsoft Teams, PagerDuty, Opsgenie or any other system that exposes a webhook interface. A complete list of integrations can be found here.

You can view and silence notifications by accessing http://<host-ip>:9093.

The notification receivers can be configured in the alertmanager/config.yml file. Before alerting works, check the Alertmanager volume mount described in Known issues.

To receive alerts via Slack, create an Incoming Webhook for your Slack workspace. You can find more details on setting up the Slack integration here.

Copy the Slack Webhook URL into the api_url field and specify a Slack channel. The shipped file contains placeholders that you must replace:

route:
    receiver: 'slack'

receivers:
    - name: 'slack'
      slack_configs:
          - send_resolved: true
            text: "{{ .CommonAnnotations.description }}"
            username: 'Prometheus'
            channel: '#<channel-name>'
            api_url: 'https://hooks.slack.com/services/<webhook-id>'

Sending metrics to the Pushgateway

The Pushgateway is used to collect data from batch jobs or from services.

To push data, simply execute:

echo "some_metric 3.14" | curl --data-binary @- http://<host-ip>:9091/metrics/job/some_job

The Pushgateway has no authentication in this stack. Prometheus scrapes it every 10 seconds with honor_labels: true, so the job and instance labels you push are kept.

Adding scrape targets

Add a job to the scrape_configs section of prometheus/prometheus.yml. Targets on the monitor-net network can be addressed by container name:

  - job_name: 'nginx'
    scrape_interval: 10s
    static_configs:
      - targets: ['nginxexporter:9113']

Then reload Prometheus as described in Define alerts. To scrape another container by name, attach it to the monitor-net network (Compose names it <project>_monitor-net, for example grafana-stack_monitor-net).

Security notes

Change the default credentials. Grafana and InfluxDB fall back to admin / admin when ADMIN_PASSWORD / INFLUX_ADMIN_PASSWORD are not set.
Most services have no authentication. Prometheus, Alertmanager, Pushgateway, cAdvisor and Node Exporter are published on all host interfaces without a login. influxdb.conf does not enable HTTP authentication. Anyone who can reach these ports can read metrics, push metrics, silence alerts, or (through the lifecycle API) reload or shut down Prometheus. Restrict the ports with a firewall, bind them to 127.0.0.1, or put a reverse proxy with authentication in front of them.
Host access. cAdvisor runs privileged and mounts /, /var/run (which includes the Docker socket), /sys and /var/lib/docker read-only. Node Exporter mounts /proc, /sys and / read-only. Only run the stack on hosts where that is acceptable.
Secrets in files. Don't commit a real Slack webhook URL in alertmanager/config.yml, or real passwords in config or .env.

Known issues

These come from the current docker-compose.yml and configuration files:

Prometheus port mapping. The compose file maps host port 9090 to container port 9000, but Prometheus listens on 9090, so http://<host-ip>:9090 does not reach Prometheus. Grafana is not affected, because it connects to http://prometheus:9090 on the internal network. Nothing sets --web.listen-address, so nothing listens on container port 9000. To expose the Prometheus UI, change the mapping to 9090:9090.
Alertmanager fails to start. Alertmanager is started with --config.file=/etc/alertmanager/config.yml, but nothing is mounted at /etc/alertmanager: the compose file mounts ./prometheus/config (which does not exist in the repository) to /etc/prometheus, and never mounts ./alertmanager/config.yml. The image only ships /etc/alertmanager/alertmanager.yml, so Alertmanager exits at startup and restarts in a loop (restart: unless-stopped), and the UI on port 9093 is unavailable. The ./prometheus/data → /prometheus mount is unused, and --storage.path=/alertmanager is not persisted. A fix is to mount ./alertmanager:/etc/alertmanager (and a data directory at /alertmanager if you want silences to survive restarts).
Storage panels and alert use aufs. Modern Docker hosts usually report ext4, xfs or btrfs, so the Storage Load / Free Storage panels and the high_storage_load alert stay empty until you change the fstype.

Troubleshooting

Grafana or Prometheus can't write to their data directory. The data directories are bind mounts. Grafana runs as user ID 472, and Prometheus runs as nobody (65534). If Docker creates the directories as root, fix the ownership and restart the stack:

sudo chown -R 472:472 ./grafana/data
sudo chown -R 65534:65534 ./prometheus_data
docker compose up -d

Upgrading from Grafana older than 5.1. In Grafana versions >= 5.1 the ID of the grafana user changed; the official image now runs as UID 472. Files created before 5.1 won't have the correct permissions for later versions:

Version	User	User ID
< 5.1	grafana	104
>= 5.1	grafana	472

There are two possible solutions to this problem:

Change the ownership from 104 to 472 (as shown above).
Start the upgraded container as user 104 by adding user: "104" to the grafana service in docker-compose.yml.

The admin password doesn't change. See Setup Grafana: the environment variables are only applied on the first start.

Dashboards show "N/A" for storage. See the fstype notes in Dashboards.

Project layout

.
├── docker-compose.yml          # all services
├── config                      # optional Grafana env file
├── alertmanager/config.yml     # Alertmanager route and receivers
├── prometheus/
│   ├── prometheus.yml          # scrape jobs, Alertmanager target
│   └── alert.rules             # alert rules
├── grafana/provisioning/
│   ├── datasources/datasource.yml
│   └── dashboards/             # dashboard.yml + dashboard JSON files
├── influxdb/influxdb.conf
└── screenshots/

Contributing

Issues and pull requests are welcome. Please keep changes to the compose file, Prometheus configuration and dashboards in separate pull requests, and describe how you tested them.

Credits

This stack is derived from dockprom by Stefan Prodan, released under the MIT License. The Docker Host, Docker Containers, Monitor Services and Nginx dashboards, the alert rules and much of this README come from that project.

Navigation Menu

Repository navigation

About

Repository files navigation

grafana-stack

Table of contents

Architecture

Services

Requirements

Install

Configuration

Environment variables

Configuration files

Prometheus scrape jobs

Setup Grafana

Dashboards

Screenshots

Define alerts

Setup alerting

Sending metrics to the Pushgateway

Adding scrape targets

Security notes

Known issues

Troubleshooting

Project layout

Contributing

Credits

License

About

Releases

Packages

Contributors

Name		Name	Last commit message	Last commit date
alertmanager		alertmanager
grafana/provisioning		grafana/provisioning
influxdb		influxdb
prometheus		prometheus
screenshots		screenshots
LICENSE		LICENSE
README.md		README.md
config		config
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation

grafana-stack

Table of contents

Architecture

Services

Requirements

Install

Configuration

Environment variables

Configuration files

Prometheus scrape jobs

Setup Grafana

Dashboards

Screenshots

Define alerts

Setup alerting

Sending metrics to the Pushgateway

Adding scrape targets

Security notes

Known issues

Troubleshooting

Project layout

Contributing

Credits

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors