Skip to content

How to debug the embedded `Grafana` in Harvester

Jump to bottom
Jian Wang edited this page Jun 16, 2023 · 1 revision

How to debug the embedded Grafana in Harvester

Background

The embedded grafana does not display well in Harveter.

https://github.com/harvester/harvester/issues/4087 [BUG]When accessing imported Harvester in Rancher, Grafana metrics don't work correctly PR: https://github.com/harvester/harvester-installer/pull/512 fix grafana dashboard 404 issue

Get grafana nginx access log

The grafana POD in cattle-monitoring-system has a container grafana-proxy, which is responsible to process those HTTP based access. To debug the issue, a first step is to get the log of it.

Following logs do not help.

harv31:~ # kubectl logs -n cattle-monitoring-system rancher-monitoring-grafana-8559b68898-qtzrz -c grafana-proxy

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/06/16 12:44:34 [warn] 1#1: duplicate MIME type "text/html" in /nginx/nginx.conf:55
nginx: [warn] duplicate MIME type "text/html" in /nginx/nginx.conf:55

Update container command

docker ps -a | grep nginx

1b440fa4ac294 docker.io/rancher/mirrored-library-nginx:1.21.1-alpine "/docker-entrypoint.…" 14 minutes ago Up k8s://cattle-monitoring-system/rancher-monitoring-grafana-7fc56c48dc-vx8x8/grafana-proxy

The grafana related POD, has command of docker-entrypoint.sh, the log only include something in this shell script.

The deployment/pod definition has only args, no command.

      - args:
        - nginx
        - -g
        - daemon off;
        - -c
        - /nginx/nginx.conf
        image: rancher/mirrored-library-nginx:1.21.1-alpine
        imagePullPolicy: IfNotPresent
        name: grafana-proxy

change the definition:

      - args:
        - -g
        - daemon off;
        - -c
        - /nginx/nginx.conf
        command:
          - nginx

Refer: https://stackoverflow.com/questions/44316361/difference-between-docker-entrypoint-and-kubernetes-container-spec-command

Enable nginx access_log

Via edit the configmap, update access_log /var/log/nginx/access.log; in server section

kk get configmap -n cattle-monitoring-system grafana-nginx-proxy-config -oyaml


apiVersion: v1
data:
  nginx.conf: |-
...

      server {
        listen          8080;
        access_log      /var/log/nginx/access.log;

With kubectl logs -n cattle-monitoring-system rancher-monitoring-grafana-67f75596fb-k7klq -c grafana-proxy, we can get log like:

192.168.122.141 - - [15/Jun/2023:08:28:25 +0000] "GET /api/dashboards/uid/rancher-cluster-1 HTTP/1.1" 200 12925 "https://192.168.122.141/dashboard/harvester/c/local/harvesterhci.io.dashboard" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"

192.168.122.141 - - [15/Jun/2023:12:48:59 +0000] "GET /api/prometheus/grafana/api/v1/rules?dashboard_uid=rancher-cluster-nodes-1 HTTP/1.1" 200 41 "https://192.168.122.159/k8s/clusters/c-m-75b2hnlw/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/d/rancher-cluster-nodes-1/rancher-cluster-nodes?orgId=1&kiosk&from=now-5m&to=now&refresh=30s&theme=light" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"

The nginx access log is roughly per below format

log_format compression '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $bytes_sent '
                       '"$http_referer" "$http_user_agent" "$gzip_ratio"';

The access log do help very much for debugging.

Debug the nginx config file

Do it on the fly via

(1) change the configmap kubectl edit configmap -n cattle-monitoring-system grafana-nginx-proxy-config

(2) replace the POD kubectl delete pod -n cattle-monitoring-system rancher-monitoring-grafana-8559b68898-qtzrz

(3) automatically, the new (replaced) grafana POD will use the configmap as nginx config file.

In the issue, our foucus is the sub_filter appSubUrl.

    location / {
...

      sub_filter '"appSubUrl":""' '"appSubUrl":"/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy"';
...

    }

Additional

There is a

rewrite ^/k8s/clusters/./proxy(.) /$1 break;

in the chart https://github.com/rancher/charts/blob/4e9cafc89224aec62bbaee7fdd5271ffbf3a0862/charts/rancher-monitoring/102.0.0%2Bup40.1.2/charts/grafana/templates/nginx-config.yaml#L84; but this seems not working, if you comment this line, it does not have any effect.

Thanks.

Clone this wiki locally