Flask Monitoring

Monitor Flask application through custom StatsD metrics with Prometheus and Grafana.

1. Flask app sends StatsD format metrics by UDP to statsd_exporter
2. Prometheus scrapes prometheus format metrics from statsd_exporter
3. Grafana queries data from Prometheus

+-----------------+                     +-------------------+                        +--------------+               +-----------+
|  Flask(StatsD)  |---(UDP repeater)--->|  statsd_exporter  |<---(scrape /metrics)---|  Prometheus  | <---(query)---|  Grafana  |
+-----------------+                     +-------------------+                        +--------------+               +-----------+

Quick Start

1. Build application image and start all service with docker-compose

   docker-compose build
   docker-compose up -d

2. Send requests with siege to flask app

   bash request-script.sh

3. Check predefined dashboard flask monitoring on Grafana http://localhost:3000/

   Dashboard screenshot:

   

   Dashboard is also available on Grafana Dashboards.

Details

Flask

1. Use Datadog Python library to send StatsD format metrics in flask_app/utils.py.
   1. setting_statsd: Register StatsD host and port with environment variable STATSD_HOST, STATSD_PORT
   2. StatsdMiddleware: Add custom metrics
2. Initial statsd with setting_statsd, and add StatsdMiddleware to flask app in flask_app/app.py.

Statsd Exporter

statsd exporter receives StatsD-style metrics and exports them as Prometheus metrics.

Ports:

1. 9125: default StatsD request listen port, send StatsD request to this port
2. 9102: Web expose port, get prometheus metrics from this port

Prometheus metrics is available on http://localhost:9102/metrics.

Prometheus

Setting scrape job of statsd-exporter with config file etc/prometheus/prometheus.yml.

scrape_configs:
  - job_name: 'statsd-exporter'
    static_configs:
      - targets: ['statsd-exporter:9102']

Prometheus build-in web UI is available on http://localhost:9090.

Grafana

Add prometheus to data source with config file etc/grafana/datasource.yml.

datasources:
  # <string, required> name of the datasource. Required
  - name: Prometheus
    # <string, required> datasource type. Required
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    # <int> org id. will default to orgId 1 if not specified
    orgId: 1
    # <string> custom UID which can be used to reference this datasource in other parts of the configuration, if not specified will be generated automatically
    # uid: my_unique_uid
    # <string> url
    url: http://prometheus:9090
    # <bool> mark as default datasource. Max one per org
    isDefault: true
    version: 1
    # <bool> allow users to edit datasources from the UI.
    editable: false

Load predefined dashboard with etc/dashboards.yaml and etc/dashboards/flask-monitoring.json.

# grafana in docker-compose.yaml
grafana:
    image: grafana/grafana:8.4.3
    volumes:
      - ./etc/grafana/:/etc/grafana/provisioning/datasources
      - ./etc/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml # dashboard setting
      - ./etc/dashboards:/etc/grafana/dashboards # dashboard json files directory

Dashboard

Flask monitoring dashboard from etc/dashboards/flask-monitoring.json

This dashboard follows RED method proposed by Weaveworks, and monitors three key metrics:

1. Rate: requests rate, request count over past 24 hours
2. Errors: 2xx rate, 5xx rate, cumulative number of each status code over past 24 hours
3. Duration: average request duration, request duration distribution (PR50, PR90, PR99)

Reference

1. Monitoring Gunicorn with Prometheus
2. Gunicorn Dashboard Sample
3. Gunircorn worker type in different scenario [Chinese]
4. Grafana datasource provisioning
5. The RED Method: key metrics for microservices architecture