The gateway Pod stuck in CrashLoopBackOff state #2976
Description
Commit range fa364ff..0e19690 (subsequent commits affected by either GH-2974 and/or GH-2975):
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
backend-7d4f996496-5ktdd 1/1 Running 0 54m
backend-cosv-65c58dc899-wlr7b 0/1 Init:0/2 0 54m
demo-5985955c49-8ct2z 1/1 Running 0 54m
demo-cpg-66d9f959f-vrbl2 1/1 Running 0 54m
frontend-cosv-565774f969-tp65l 1/1 Running 0 54m
frontend-df5ccd8fc-8wfnj 1/1 Running 0 54m
gateway-7ccfdb8788-kbcwg 0/1 CrashLoopBackOff 15 54m
loki-canary-4v8q8 1/1 Running 0 54m
loki-canary-qcttf 1/1 Running 0 54m
loki-canary-xv6zm 1/1 Running 0 54m
mysql-tunnel 1/1 Running 0 45h
orchestrator-56568c8cdf-drxxl 1/1 Running 0 54m
preprocessor-5cd447949b-9ngw4 1/1 Running 0 54m
save-cloud-0 1/1 Running 0 54m
save-cloud-grafana-agent-operator-6b9d4f9d8d-qsrxr 1/1 Running 0 54m
save-cloud-grafana-c8c645d67-shc6j 1/1 Running 0 54m
save-cloud-loki-0 1/1 Running 0 54m
save-cloud-prometheus-server-6f8577bbf6-9sgjn 1/1 Running 0 54m
save-cloud-promtail-df95d 1/1 Running 0 54m
save-cloud-promtail-fzp28 1/1 Running 0 54m
save-cloud-promtail-kvg6q 1/1 Running 0 54mkubectl describe for backend-cosv-65c58dc899-wlr7b shows the following events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 56m default-scheduler Successfully assigned save-cloud/backend-cosv-65c58dc899-wlr7b to 172.16.0.55
Warning FailedMount 49m kubelet Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[database-secret ca-certs config-volume s3-secrets migrations-data kube-api-access-ztcpw]: timed out waiting for the condition
Warning FailedMount 44m (x3 over 53m) kubelet Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[migrations-data kube-api-access-ztcpw database-secret ca-certs config-volume s3-secrets]: timed out waiting for the condition
Warning FailedMount 38m kubelet Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[ca-certs config-volume s3-secrets migrations-data kube-api-access-ztcpw database-secret]: timed out waiting for the condition
Warning FailedMount 20m (x3 over 47m) kubelet Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[s3-secrets migrations-data kube-api-access-ztcpw database-secret ca-certs config-volume]: timed out waiting for the condition
Warning FailedMount 15m (x3 over 40m) kubelet Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[kube-api-access-ztcpw database-secret ca-certs config-volume s3-secrets migrations-data]: timed out waiting for the condition
Warning FailedMount 10m (x5 over 42m) kubelet Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[config-volume s3-secrets migrations-data kube-api-access-ztcpw database-secret ca-certs]: timed out waiting for the condition
Warning FailedMount 61s (x35 over 56m) kubelet MountVolume.SetUp failed for volume "cacerts" : configmap "ca-certs" not found
Events for gateway-7ccfdb8788-kbcwg (0.4.0-alpha.0.429-0e19690):
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 58m default-scheduler Successfully assigned save-cloud/gateway-7ccfdb8788-kbcwg to 172.16.0.87
Normal Pulled 58m kubelet Successfully pulled image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690" in 3.485966887s
Normal SuccessfulMountVolume 58m (x2 over 58m) kubelet Successfully mounted volumes for pod "gateway-7ccfdb8788-kbcwg_save-cloud(d2bb10f3-1375-45ee-b550-fbf3a69661a6)"
Normal Pulled 58m kubelet Successfully pulled image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690" in 984.39591ms
Normal Pulling 58m (x3 over 58m) kubelet Pulling image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690"
Normal SuccessfulCreate 58m (x3 over 58m) kubelet Created container gateway
Normal Started 58m (x3 over 58m) kubelet Started container gateway
Warning BackOffStart 58m (x5 over 58m) kubelet Back-off restarting failed container
Normal Pulled 58m kubelet Successfully pulled image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690" in 943.128249ms
Warning BackOffStart 3m20s (x278 over 58m) kubelet the failed container exited with ExitCode: 82
Events for gateway-8454c8bcb6-hbrhs (0.4.0-alpha.0.414-fa364ff):
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m57s default-scheduler Successfully assigned save-cloud/gateway-8454c8bcb6-hbrhs to 172.16.0.52
Warning FailedMount 2m54s kubelet Unable to attach or mount volumes: unmounted volumes=[his-certs], unattached volumes=[cacerts his-certs kube-api-access-4fv7f config-volume oauth-credentials]: timed out waiting for the condition
Warning FailedMount 47s (x10 over 4m57s) kubelet MountVolume.SetUp failed for volume "his-certs" : secret "his-certs" not found
Warning FailedMount 38s kubelet Unable to attach or mount volumes: unmounted volumes=[his-certs], unattached volumes=[his-certs kube-api-access-4fv7f config-volume oauth-credentials cacerts]: timed out waiting for the condition
Logs for backend-cosv-65c58dc899-wlr7b:
Defaulted container "backend-cosv" out of: backend-cosv, git-cloner (init), liquibase-runner (init)
Error from server (BadRequest): container "backend-cosv" in pod "backend-cosv-65c58dc899-wlr7b" is waiting to start: PodInitializing
Logs for gateway-7ccfdb8788-kbcwg (0.4.0-alpha.0.429-0e19690):
Setting Active Processor Count to 4
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx91737K -XX:MaxMetaspaceSize=188294K -Xss1M (Total Memory: 390624K, Thread Count: 50, Loaded Class Count: 30830, Headroom: 0
%)
Enabling Java Native Memory Tracking
unable to load certificates
Using readonly truststore: /tmp/truststore
unable to read keystore
unable to open /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
open /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts: no such file or directory
ERROR: failed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader': exit status 1
- Caused by: fa364ff, updated deployments config for HIS usecase #2898
- Helm chart version range:
0.2.1+1181..0.2.1+1196 - Date range: 29.12.2023..31.01.2024
- Commit range: fa364ff..0e19690
- Docker tag range:
0.4.0-alpha.0.414-fa364ff..0.4.0-alpha.0.429-0e19690