Monitoring for cht-user-management instances

[kennsippell]

UMT added prometheus endpoint medic/cht-user-management#68 like https://users-chis-ke.app.medicmobile.org/metrics. How shall watchdog now scrape these?

[kennsippell]

@mrjones-plip @jkuester I'd be interested in your thoughts on how to approach this.

Should Watchdog be expanded to monitor non-CHT services like UMT? If so, would we add something like umt-instances.yml alongside cht-instances.yml?

[jkuester]

IMHO it makes sense to include optional config in the cht-watchdog repo for connecting to various other systems that are commonly run in the CHT ecosystem (e.g. like what we currently have for cht-sync). The UMT seems like a perfect example of this! 👍

I have not looked at this too closely, but my initial though regarding the implementation details would be to follow what we did for cht-sync and have a separate directory in ./exporters which contains any config (compose files, etc) needed to make things work.

[mrjones-plip]

100% agree with @jkuester !

@kennsippell - I'm avail to help write docs for this or help test yaml config files for watchdog.

[kennsippell]

@mrjones-plip There is a pull request at #146 which I'd love your input and testing on. It adds UMT as an "additional config" and adds a new stock dashboard. After merge, I'd appreciate your help turning this on for https://watchdog.app.medicmobile.org

[mrjones-plip]

Awesome - I'll get to this in the next day or two!

[mrjones-plip]

I'm stuck on basic setup! Can you have a looksee at these steps and see what I'm doing wrong? (don't worry about that bug just now) Also posted over in Slack, but feel free to reply here - this is more public.

[mrjones-plip]

Now that #146 and medic/cht-user-management#278 are merged, CI/CD ran and all 4 prod UMT instances now have a metrics endpoint and prod watchdog should have configs ready to use. I'll add these 4 URLs to prod watchdog!

• users-chis-ml.app.medicmobile.org/metrics
• users-chis-civ.app.medicmobile.org/metrics
• users-chis-tg.app.medicmobile.org/metrics
• users-chis-ke.app.medicmobile.org/metrics

I had some issues, but I've got watchdog prod on latest - will work on the rest soon!

[mrjones-plip]

Steps to deploy:

1. copy scrape conf: cp /root/cht-monitoring/exporters/cht-user-management/scrape.yml /root/cht-monitoring/exporters/cht-user-management/scrape-custom.yml and add values

    root@watchdog:~/cht-monitoring# cat exporters/cht-user-management/scrape-custom.yml 
    scrape_configs:
      - job_name: cht-user-management-metrics
        scheme: "https"
        static_configs:
          - targets:
    	  - users-chis-ml.app.medicmobile.org 
    	  - users-chis-civ.app.medicmobile.org 
    	  - users-chis-tg.app.medicmobile.org 
    	  - users-chis-ke.app.medicmobile.org
   

2. add new compose file to /root/down-up.sh script: ./exporters/cht-user-management/compose.yml

    root@watchdog:~# cat /root/down-up.sh 
    #!/bin/bash
    cd ~/cht-monitoring
    docker compose \
    	-f docker-compose.yml \
    	-f exporters/postgres/compose.yml \
    	-f ../caddy-compose.yml \
    	-f ../docker-compose-cht3x.yml \
    	-f data-ingest/extra-sql-compose.yml \
    	-f node-exporter/compose.yml \
    	-f exporters/cht-user-management/compose.yml \
    	down 
   
    docker compose \
    	-f docker-compose.yml \
    	-f exporters/postgres/compose.yml \
    	-f ../caddy-compose.yml \
    	-f ../docker-compose-cht3x.yml \
    	-f data-ingest/extra-sql-compose.yml \
    	-f node-exporter/compose.yml \
    	-f exporters/cht-user-management/compose.yml \
    	up --remove-orphans -d
   

3. run down up

    root@watchdog:~/cht-monitoring# /root/down-up.sh 
    [+] Running 7/7
     ✔ Container cht-monitoring-prometheus-1          Removed                                                                               0.6s 
     ✔ Container cht-monitoring-grafana-1             Removed                                                                               0.6s 
     ✔ Container cht-monitoring-sql_exporter-1        Removed                                                                               0.6s 
     ✔ Container cht-monitoring-extra_sql_exporter-1  Removed                                                                               0.5s 
     ✔ Container cht-monitoring-json-exporter-1       Removed                                                                               0.3s 
     ✔ Container cht-monitoring-caddy-1               Removed                                                                               0.5s 
     ✔ Network cht-monitoring_cht-watchdog-net        Removed                                                                               0.1s 
    [+] Running 7/7
     ✔ Network cht-monitoring_cht-watchdog-net        Created                                                                               0.1s 
     ✔ Container cht-monitoring-extra_sql_exporter-1  Started                                                                               1.0s 
     ✔ Container cht-monitoring-grafana-1             Started                                                                               1.1s 
     ✔ Container cht-monitoring-json-exporter-1       Started                                                                               0.9s 
     ✔ Container cht-monitoring-prometheus-1          Started                                                                               1.0s 
     ✔ Container cht-monitoring-sql_exporter-1        Started                                                                               1.1s 
     ✔ Container cht-monitoring-caddy-1               Started                                                                               0.8s 
    root@watchdog:~/cht-monitoring# 
   

Logging into watchdog it looks good!