Looking at the events it looks like it was prempted (longer logs below) extract here. It looks like scheduling failed and then scale-up faiiled/was not triggered and max group size reached.

55m         Normal    Preempted                pod/support-prometheus-server-548d55b8bc-hfkvk    Preempted by kube-system/calico-node-rnths on node gke-leap-cluster-core-pool-1cc6bf7d-g77w
55m         Normal    Killing                  pod/support-prometheus-server-548d55b8bc-hfkvk    Stopping container prometheus-server-configmap-reload
55m         Normal    Killing                  pod/support-prometheus-server-548d55b8bc-hfkvk    Stopping container prometheus-server
55m         Warning   FailedScheduling         pod/support-prometheus-server-548d55b8bc-mhppb    0/212 nodes are available: 200 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 5 Insufficient cpu, 7 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 8 Insufficient memory. preemption: 0/212 nodes are available: 207 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod.
55m         Normal    NotTriggerScaleUp        pod/support-prometheus-server-548d55b8bc-mhppb    pod didn't trigger scale-up: 4 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 2 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 2 Insufficient memory, 1 Insufficient cpu, 2 max node group size reached

k get events -n support

LAST SEEN   TYPE      REASON                   OBJECT                                            MESSAGE
26m         Normal    Scheduled                pod/support-cryptnono-6rmkq                       Successfully assigned support/support-cryptnono-6rmkq to gke-leap-cluster-core-pool-1cc6bf7d-tmq0
26m         Warning   FailedCreatePodSandBox   pod/support-cryptnono-6rmkq                       Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f53c22cf29bd154c5839d963af765266c5c2076f94b490a251100ab3693f5c7e": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
26m         Normal    Pulling                  pod/support-cryptnono-6rmkq                       Pulling image "yuvipanda/kubectl-trace-init:9af18f1197b1b377be4816c2761049745b64c6b8"
26m         Normal    Pulled                   pod/support-cryptnono-6rmkq                       Successfully pulled image "yuvipanda/kubectl-trace-init:9af18f1197b1b377be4816c2761049745b64c6b8" in 6.317139676s
26m         Normal    Created                  pod/support-cryptnono-6rmkq                       Created container kubectl-trace-init
26m         Normal    Started                  pod/support-cryptnono-6rmkq                       Started container kubectl-trace-init
24m         Normal    Pulling                  pod/support-cryptnono-6rmkq                       Pulling image "quay.io/iovisor/bpftrace:1c295d899035aaded34d5861375051ec1add1d3a-vanilla_llvm12_clang_glibc2.23"
24m         Normal    Pulled                   pod/support-cryptnono-6rmkq                       Successfully pulled image "quay.io/iovisor/bpftrace:1c295d899035aaded34d5861375051ec1add1d3a-vanilla_llvm12_clang_glibc2.23" in 3.724225512s
24m         Normal    Created                  pod/support-cryptnono-6rmkq                       Created container trace
24m         Normal    Started                  pod/support-cryptnono-6rmkq                       Started container trace
18m         Normal    Killing                  pod/support-cryptnono-jxqgh                       Stopping container trace
18m         Warning   FailedPreStopHook        pod/support-cryptnono-jxqgh                       Exec lifecycle hook ([/bin/bash -c kill -SIGINT $(pidof bpftrace) && sleep 30]) for Container "trace" in Pod "support-cryptnono-jxqgh_support(668e877f-e240-454f-b701-969e684aaa12)" failed - error: command '/bin/bash -c kill -SIGINT $(pidof bpftrace) && sleep 30' exited with 137: , message: ""
16m         Normal    Scheduled                pod/support-cryptnono-k9lvj                       Successfully assigned support/support-cryptnono-k9lvj to gke-leap-cluster-dask-huge-b3a90f26-574h
16m         Warning   FailedCreatePodSandBox   pod/support-cryptnono-k9lvj                       Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "68bd292ac298b53878944d07d1fdb1a4b221ab7b2a4269474d2c80ffe787bb88": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
16m         Normal    Pulling                  pod/support-cryptnono-k9lvj                       Pulling image "yuvipanda/kubectl-trace-init:9af18f1197b1b377be4816c2761049745b64c6b8"
16m         Normal    Pulled                   pod/support-cryptnono-k9lvj                       Successfully pulled image "yuvipanda/kubectl-trace-init:9af18f1197b1b377be4816c2761049745b64c6b8" in 4.904876655s
16m         Normal    Created                  pod/support-cryptnono-k9lvj                       Created container kubectl-trace-init
16m         Normal    Started                  pod/support-cryptnono-k9lvj                       Started container kubectl-trace-init
15m         Normal    Pulling                  pod/support-cryptnono-k9lvj                       Pulling image "quay.io/iovisor/bpftrace:1c295d899035aaded34d5861375051ec1add1d3a-vanilla_llvm12_clang_glibc2.23"
15m         Normal    Pulled                   pod/support-cryptnono-k9lvj                       Successfully pulled image "quay.io/iovisor/bpftrace:1c295d899035aaded34d5861375051ec1add1d3a-vanilla_llvm12_clang_glibc2.23" in 3.016818465s
15m         Normal    Created                  pod/support-cryptnono-k9lvj                       Created container trace
15m         Normal    Started                  pod/support-cryptnono-k9lvj                       Started container trace
57m         Normal    Scheduled                pod/support-cryptnono-pqfxr                       Successfully assigned support/support-cryptnono-pqfxr to gke-leap-cluster-nb-large-3ab858ce-gmz9
57m         Warning   FailedCreatePodSandBox   pod/support-cryptnono-pqfxr                       Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e87524ae545827b1bb95712db0b0d42131f31faf0bb35d58f9cc798d40c46b41": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
57m         Normal    Pulling                  pod/support-cryptnono-pqfxr                       Pulling image "yuvipanda/kubectl-trace-init:9af18f1197b1b377be4816c2761049745b64c6b8"
53m         Normal    Pulled                   pod/support-cryptnono-pqfxr                       Successfully pulled image "yuvipanda/kubectl-trace-init:9af18f1197b1b377be4816c2761049745b64c6b8" in 3m50.935963258s
53m         Normal    Created                  pod/support-cryptnono-pqfxr                       Created container kubectl-trace-init
53m         Normal    Started                  pod/support-cryptnono-pqfxr                       Started container kubectl-trace-init
51m         Normal    Pulling                  pod/support-cryptnono-pqfxr                       Pulling image "quay.io/iovisor/bpftrace:1c295d899035aaded34d5861375051ec1add1d3a-vanilla_llvm12_clang_glibc2.23"
49m         Normal    Pulled                   pod/support-cryptnono-pqfxr                       Successfully pulled image "quay.io/iovisor/bpftrace:1c295d899035aaded34d5861375051ec1add1d3a-vanilla_llvm12_clang_glibc2.23" in 1m29.808725295s
49m         Normal    Created                  pod/support-cryptnono-pqfxr                       Created container trace
49m         Normal    Started                  pod/support-cryptnono-pqfxr                       Started container trace
57m         Normal    SuccessfulCreate         daemonset/support-cryptnono                       (combined from similar events): Created pod: support-cryptnono-pqfxr
26m         Normal    SuccessfulCreate         daemonset/support-cryptnono                       Created pod: support-cryptnono-6rmkq
18m         Warning   FailedDaemonPod          daemonset/support-cryptnono                       Found failed daemon pod support/support-cryptnono-jxqgh on node gke-leap-cluster-dask-huge-b3a90f26-574h, will try to kill it
18m         Normal    SuccessfulDelete         daemonset/support-cryptnono                       Deleted pod: support-cryptnono-jxqgh
16m         Normal    SuccessfulCreate         daemonset/support-cryptnono                       Created pod: support-cryptnono-k9lvj
16m         Normal    UpdatedLoadBalancer      service/support-ingress-nginx-controller          Updated load balancer with new hosts
26m         Normal    Scheduled                pod/support-prometheus-node-exporter-bzh9k        Successfully assigned support/support-prometheus-node-exporter-bzh9k to gke-leap-cluster-core-pool-1cc6bf7d-tmq0
26m         Normal    Pulling                  pod/support-prometheus-node-exporter-bzh9k        Pulling image "quay.io/prometheus/node-exporter:v1.5.0"
26m         Normal    Pulled                   pod/support-prometheus-node-exporter-bzh9k        Successfully pulled image "quay.io/prometheus/node-exporter:v1.5.0" in 11.452735399s
26m         Normal    Created                  pod/support-prometheus-node-exporter-bzh9k        Created container node-exporter
26m         Normal    Started                  pod/support-prometheus-node-exporter-bzh9k        Started container node-exporter
16m         Normal    Scheduled                pod/support-prometheus-node-exporter-hnhmf        Successfully assigned support/support-prometheus-node-exporter-hnhmf to gke-leap-cluster-dask-huge-b3a90f26-574h
16m         Normal    Pulling                  pod/support-prometheus-node-exporter-hnhmf        Pulling image "quay.io/prometheus/node-exporter:v1.5.0"
16m         Normal    Pulled                   pod/support-prometheus-node-exporter-hnhmf        Successfully pulled image "quay.io/prometheus/node-exporter:v1.5.0" in 4.595472136s
16m         Normal    Created                  pod/support-prometheus-node-exporter-hnhmf        Created container node-exporter
16m         Normal    Started                  pod/support-prometheus-node-exporter-hnhmf        Started container node-exporter
18m         Normal    Killing                  pod/support-prometheus-node-exporter-pgd4t        Stopping container node-exporter
57m         Normal    Scheduled                pod/support-prometheus-node-exporter-s967n        Successfully assigned support/support-prometheus-node-exporter-s967n to gke-leap-cluster-nb-large-3ab858ce-gmz9
57m         Normal    Pulling                  pod/support-prometheus-node-exporter-s967n        Pulling image "quay.io/prometheus/node-exporter:v1.5.0"
57m         Normal    Pulled                   pod/support-prometheus-node-exporter-s967n        Successfully pulled image "quay.io/prometheus/node-exporter:v1.5.0" in 5.175019332s
57m         Normal    Created                  pod/support-prometheus-node-exporter-s967n        Created container node-exporter
57m         Normal    Started                  pod/support-prometheus-node-exporter-s967n        Started container node-exporter
57m         Normal    SuccessfulCreate         daemonset/support-prometheus-node-exporter        (combined from similar events): Created pod: support-prometheus-node-exporter-s967n
26m         Normal    SuccessfulCreate         daemonset/support-prometheus-node-exporter        Created pod: support-prometheus-node-exporter-bzh9k
18m         Warning   FailedDaemonPod          daemonset/support-prometheus-node-exporter        Found failed daemon pod support/support-prometheus-node-exporter-pgd4t on node gke-leap-cluster-dask-huge-b3a90f26-574h, will try to kill it
18m         Normal    SuccessfulDelete         daemonset/support-prometheus-node-exporter        Deleted pod: support-prometheus-node-exporter-pgd4t
16m         Normal    SuccessfulCreate         daemonset/support-prometheus-node-exporter        Created pod: support-prometheus-node-exporter-hnhmf
57m         Normal    Preempted                pod/support-prometheus-server-548d55b8bc-hfkvk    Preempted by kube-system/calico-node-rnths on node gke-leap-cluster-core-pool-1cc6bf7d-g77w
57m         Normal    Killing                  pod/support-prometheus-server-548d55b8bc-hfkvk    Stopping container prometheus-server-configmap-reload
57m         Normal    Killing                  pod/support-prometheus-server-548d55b8bc-hfkvk    Stopping container prometheus-server
57m         Warning   FailedScheduling         pod/support-prometheus-server-548d55b8bc-mhppb    0/212 nodes are available: 200 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 5 Insufficient cpu, 7 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 8 Insufficient memory. preemption: 0/212 nodes are available: 207 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod.
57m         Normal    NotTriggerScaleUp        pod/support-prometheus-server-548d55b8bc-mhppb    pod didn't trigger scale-up: 4 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 2 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 2 Insufficient memory, 1 Insufficient cpu, 2 max node group size reached
52m         Normal    NotTriggerScaleUp        pod/support-prometheus-server-548d55b8bc-mhppb    pod didn't trigger scale-up: 3 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 5 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 2 max node group size reached
54m         Warning   FailedScheduling         pod/support-prometheus-server-548d55b8bc-mhppb    0/212 nodes are available: 200 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 5 Insufficient cpu, 7 Insufficient memory, 7 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}. preemption: 0/212 nodes are available: 207 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod.
36m         Normal    NotTriggerScaleUp        pod/support-prometheus-server-548d55b8bc-mhppb    pod didn't trigger scale-up: 5 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 3 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 2 max node group size reached
31m         Normal    NotTriggerScaleUp        pod/support-prometheus-server-548d55b8bc-mhppb    (combined from similar events): pod didn't trigger scale-up: 2 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 1 Insufficient cpu, 1 Insufficient memory, 2 max node group size reached, 5 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}
29m         Warning   FailedScheduling         pod/support-prometheus-server-548d55b8bc-mhppb    0/212 nodes are available: 11 Insufficient cpu, 200 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 7 Insufficient memory, 7 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}. preemption: 0/212 nodes are available: 207 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod.
26m         Warning   FailedScheduling         pod/support-prometheus-server-548d55b8bc-mhppb    0/213 nodes are available: 1 node(s) had volume node affinity conflict, 11 Insufficient cpu, 200 node(s) had untolerated taint {k8s.dask.org_dedicated: worker}, 7 Insufficient memory, 7 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}. preemption: 0/213 nodes are available: 208 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod.
26m         Normal    SuccessfulAttachVolume   pod/support-prometheus-server-548d55b8bc-mhppb    AttachVolume.Attach succeeded for volume "pvc-9f8c30e3-b63f-4c82-a819-80ea211e77fe"
26m         Normal    Pulling                  pod/support-prometheus-server-548d55b8bc-mhppb    Pulling image "jimmidyson/configmap-reload:v0.8.0"
26m         Normal    Pulled                   pod/support-prometheus-server-548d55b8bc-mhppb    Successfully pulled image "jimmidyson/configmap-reload:v0.8.0" in 950.407348ms
26m         Normal    Created                  pod/support-prometheus-server-548d55b8bc-mhppb    Created container prometheus-server-configmap-reload
26m         Normal    Started                  pod/support-prometheus-server-548d55b8bc-mhppb    Started container prometheus-server-configmap-reload
26m         Normal    Pulling                  pod/support-prometheus-server-548d55b8bc-mhppb    Pulling image "quay.io/prometheus/prometheus:v2.41.0"
26m         Normal    Pulled                   pod/support-prometheus-server-548d55b8bc-mhppb    Successfully pulled image "quay.io/prometheus/prometheus:v2.41.0" in 5.241799705s
26m         Normal    Created                  pod/support-prometheus-server-548d55b8bc-mhppb    Created container prometheus-server
26m         Normal    Started                  pod/support-prometheus-server-548d55b8bc-mhppb    Started container prometheus-server
24m         Warning   Unhealthy                pod/support-prometheus-server-548d55b8bc-mhppb    Readiness probe failed: HTTP probe failed with statuscode: 503
57m         Normal    SuccessfulCreate         replicaset/support-prometheus-server-548d55b8bc   Created pod: support-prometheus-server-548d55b8bc-mhppb

LEAP prometheus server is down/scheduler faiiling #2248

Description

pnasratopenedon Feb 23, 2023

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

pnasrat
openedon Feb 23, 2023