모니터링 환경 구축

[junha-ahn]

Description

프로메테우스 그라파나 조합을 사용해서 모니터링 환경을 구축합니다.

사용중인 AWS 모든 리소스를 대상으로 한다

• EKS
• RDS
• ALB/NLB

To do

• Terraform을 통한 설치 자동화 (helm_provider 등)
  • 프로메테우스, Grafana
• Grafana 대시보드 적용 (grafana dashboard import)
• Terraform을 통한 AWS RDS 모니터링 (프로메테우스 exporter 설치 등)
• Terraform을 통한 ALB 모니터링 (프로메테우스 exporter 설치 등)
• Grafana 외부 접속 가능화 (Ingress)
• Grafana 계정 컨트롤 (후순위)
  • 계정 비밀번호 k8s secret 화 (현재 Default 사용)
  • 권한에 따른 계정 Role 컨트롤 (Admin, Developer)

연관 이슈 (해당 이슈에서는 미진행합니다)

• prometheus-grafana AWS EBS 연결 #40
• Spring App 모니터링 (Spring Boot Actuator) #44

Test Checklist

Grafana를 통해 모니터링 성공

• 각 POD의 CPU usage
• ALB의 요청량
• RDS의 CPU usage, Now Connection Count
• 스프링 부트 프로세스 Memory Usage, JVM memory Usage

[junha-ahn]

kube-prometheus-stack

참고

• https://faun.pub/kube-prometheus-with-terraform-and-helm-24cac706fbdb
• values: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

오류: Node 부족

Node 부족으로 인한 문제 발생

• Node 개수 증가로 해결

resource "helm_release" "kube-prometheus" {
  chart      = "kube-prometheus-stack"
  name       = "prometheus"
  namespace  = "monitoring"
  repository = "https://prometheus-community.github.io/helm-charts"
}

  # helm_release.kube-prometheus will be created
  + resource "helm_release" "kube-prometheus" {
      + atomic                     = false
      + chart                      = "kube-prometheus-stack"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = false
      + id                         = (known after apply)
      + lint                       = false
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "prometheus"
      + namespace                  = "monitoring"
      + pass_credentials           = false
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://prometheus-community.github.io/helm-charts"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + verify                     = false
      + version                    = "48.4.0"
      + wait                       = true
      + wait_for_jobs              = false
    }

Plan: 1 to add, 0 to change, 0 to destroy.

│ Warning: Helm release "prometheus" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│ 
│   with helm_release.kube-prometheus,
│   on monitoring.tf line 1, in resource "helm_release" "kube-prometheus":
│    1: resource "helm_release" "kube-prometheus" {
│ 
╵
╷
│ Error: failed post-install: 1 error occurred:
│       * timed out waiting for the condition
│ 
│ 
│ 
│   with helm_release.kube-prometheus,
│   on monitoring.tf line 1, in resource "helm_release" "kube-prometheus":
│    1: resource "helm_release" "kube-prometheus" {
│ 
╵
Releasing state lock. This may take a few moments...



$ kubectl get pods -n=monitoring
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          71s
prometheus-grafana-7886dd7b76-2664b                      3/3     Running   0          73s
prometheus-kube-prometheus-admission-patch-vngr4         0/1     Pending   0          44s
prometheus-kube-prometheus-operator-58fb95695d-6mmwk     1/1     Running   0          73s
prometheus-kube-state-metrics-7f4f499cb5-kmvkg           1/1     Running   0          73s
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          71s
prometheus-prometheus-node-exporter-j4qhg                1/1     Running   0          73s
prometheus-prometheus-node-exporter-n5b5r                1/1     Running   0          73s

• prometheus-kube-prometheus-admission-patch Pending

$ kubectl describe pod prometheus-kube-prometheus-admission-patch-vngr4 -n=monitoring
...

  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  86s   default-scheduler  0/2 nodes are available: 2 Too many pods. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod..

[junha-ahn]

프로메테우스 접속

$ kubectl port-forward svc/prometheus-kube-prometheus-prometheus 3300:9090 --namespace monitoring

Grafana 접속

$ kubectl get svc --namespace monitoring
NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   3m33s
prometheus-grafana                        ClusterIP   172.20.1.99      <none>        80/TCP                       3m35s
prometheus-kube-prometheus-alertmanager   ClusterIP   172.20.66.88     <none>        9093/TCP,8080/TCP            3m35s
prometheus-kube-prometheus-operator       ClusterIP   172.20.240.0     <none>        443/TCP                      3m35s
prometheus-kube-prometheus-prometheus     ClusterIP   172.20.133.170   <none>        9090/TCP,8080/TCP            3m35s
prometheus-kube-state-metrics             ClusterIP   172.20.116.253   <none>        8080/TCP                     3m35s
prometheus-operated                       ClusterIP   None             <none>        9090/TCP                     3m33s
prometheus-prometheus-node-exporter       ClusterIP   172.20.60.222    <none>        9100/TCP                     3m35s


$ kubectl port-forward svc/prometheus-grafana 3000:80 --namespace monitoring
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

$ kubectl -n monitoring get secret prometheus-grafana -o jsonpath="{.data.admin-user}" | base64 -d; echo

$ kubectl -n monitoring get secret prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 -d; echo

[hihahayoung]

MySQL Exporter

관련 명령어

kubectl port-forward svc/mysql-exporter-prometheus-mysql-exporter 3400:9104 --namespace monitoring

 $ kubectl exec -it mysql-exporter-prometheus-mysql-exporter-5c54b7d6dd-7zhjg -n=monitoring -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
...

POD

$ kubectl describe pod mysql-exporter-prometheus-mysql-exporter-5c54b7d6dd-7zhjg -n=monitoring
Name:             mysql-exporter-prometheus-mysql-exporter-5c54b7d6dd-7zhjg
Namespace:        monitoring
Priority:         0
Service Account:  default
Node:             ip-10-0-93-81.ap-northeast-2.compute.internal/10.0.93.81
Start Time:       Sun, 10 Sep 2023 08:00:13 +0000
Labels:           app.kubernetes.io/instance=mysql-exporter
                  app.kubernetes.io/name=prometheus-mysql-exporter
                  pod-template-hash=5c54b7d6dd
Annotations:      checksum/credentials: 4b661fd4f7b4c9cff26ffce2973f62bf53f2ff5480dcfbeaa52991720b816ea5
                  prometheus.io/path: /metrics
                  prometheus.io/port: 9104
                  prometheus.io/scrape: true
Status:           Running
IP:               10.0.91.16
IPs:
  IP:           10.0.91.16
Controlled By:  ReplicaSet/mysql-exporter-prometheus-mysql-exporter-5c54b7d6dd
Containers:
  prometheus-mysql-exporter:
    Container ID:  containerd://fffcab003602b14bf0c9be38c3e038ef770fff63014f7936c5ccce9b49658e74
    Image:         quay.io/prometheus/mysqld-exporter:v0.15.0
    Image ID:      quay.io/prometheus/mysqld-exporter@sha256:6b693c6c003bf51ffc2305f3d1a35d16da678bf421bfccca48ecc6077073634e
    Port:          9104/TCP
    Host Port:     0/TCP
    Args:
      --config.my-cnf=/etc/cnf/my.cnf
    State:          Running
      Started:      Sun, 10 Sep 2023 08:00:13 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9104/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9104/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /etc/cnf/ from secret-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-46p97 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  secret-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  mysql-exporter-prometheus-mysql-exporter-config
    Optional:    false
  kube-api-access-46p97:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

문제상황 (MySQL 접속 불가능)

$ kubectl logs -f mysql-exporter-prometheus-mysql-exporter-5c54b7d6dd-7zhjg -n=monitoring
ts=2023-09-10T08:00:13.828Z caller=mysqld_exporter.go:220 level=info msg="Starting mysqld_exporter" version="(version=0.15.0, branch=HEAD, revision=6ca2a42f97f3403c7788ff4f374430aa267a6b6b)"
ts=2023-09-10T08:00:13.829Z caller=mysqld_exporter.go:221 level=info msg="Build context" build_context="(go=go1.20.5, platform=linux/amd64, user=root@c4fca471a5b1, date=20230624-04:09:04, tags=netgo)"
ts=2023-09-10T08:00:13.829Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=global_status
ts=2023-09-10T08:00:13.829Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=global_variables
ts=2023-09-10T08:00:13.829Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=slave_status
ts=2023-09-10T08:00:13.829Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=info_schema.innodb_cmp
ts=2023-09-10T08:00:13.830Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=info_schema.innodb_cmpmem
ts=2023-09-10T08:00:13.830Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=info_schema.query_response_time
ts=2023-09-10T08:00:13.830Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9104
ts=2023-09-10T08:00:13.830Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9104
ts=2023-09-10T08:00:35.813Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="Error 1049 (42000): Unknown database '&lock_wait_timeout=2'"

Config

kubectl get secrets/mysql-exporter-prometheus-mysql-exporter-config -n=monitoring -o jsonpath="{.data}" 

base64 decode 이후

[client]
host=ticketingdb.c0rf7nfpwcj4.ap-northeast-2.rds.amazonaws.com
port=3306
user=root
password=m[?ntOsHLZqd9HqiT6(6mg?(MeY}
database=ticketingdb
protocol=TCP

해결

CREATE USER 'exporter'@'%' IDENTIFIED BY 'pw1234' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';

MySQL root의 비밀번호는 특수문자이기때문에 발생한 이슈

새로운 유저를 생성해서 해결

mysql:
  db: "ticketingdb"
  host: "ticketingdb.c0rf7nfpwcj4.ap-northeast-2.rds.amazonaws.com"
  port: 3306
  protocol: "TCP"
-  user: "root"
+  user: "exporter"
- pass: "m[?ntOsHLZqd9HqiT6(6mg?(MeY}"
+  pass: "pw1234"