[Flaking Test] [EventedPLEG] Containers Lifecycle should continue running liveness probes for restartable init containers and restart them while in preStop

[pacoxu]

Which jobs are flaking?

ci-crio-cgroupv1-evented-pleg

• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-crio-cgroupv1-evented-pleg/1834117609461649408

Which tests are flaking?

E2eNode Suite.[It] [sig-node] [NodeConformance] Containers Lifecycle when a pod is terminating because its liveness probe fails should continue running liveness probes for restartable init containers and restart them while in preStop [NodeConformance]

Since when has it been flaking?

8/24

https://storage.googleapis.com/k8s-triage/index.html?date=2024-09-12&job=ci-crio-cgroupv1-evented-pleg&test=%20Containers%20Lifecycle%20when%20a%20pod%20is%20terminating%20because%20its%20liveness%20probe%20fails%20should%20continue%20running%20liveness%20probes%20for%20restartable%20init%20containers%20and%20restart%20them%20while%20in%20preStop%20

Testgrid link

https://testgrid.k8s.io/sig-node-cri-o#ci-crio-cgroupv1-evented-pleg

Reason for failure (if possible)

{ failed [FAILED] Expected an error to have occurred.  Got:
    <nil>: nil
In [It] at: k8s.io/kubernetes/test/e2e_node/container_lifecycle_test.go:903 @ 08/23/24 18:16:25.131
}

Anything else we need to know?

No response

Relevant SIG(s)

/sig node

[pacoxu]

/cc @hshiina @SergeyKanzhelev

[pacoxu]

@hshiina this seem to be the same problem of #123087 static pod when EventedPLEG is enabled.)

• Fix evented pleg mirror pod & use IsEventedPLEGInUse instead of FG status check #122778 (comment) has a reproduce method.
• Restart the init container to not be stuck in created state #126543 fixed a Generic PLEG regression.

/cc @gjkim42 @liggitt

[hshiina]

As far as I saw the log, containers does not look to have been recreated.

If I understand correctly, this test works like:

1. The liveness probe for the regular container fails.
2. kubelet starts to stop the regular container. Then, the prestop hook is triggered.
3. If the liveness probe for the sidecar container runs and fails before the prestop, this assertion is passed. If the probe runs while the prostop is running, this assertion fails:
   

   kubernetes/test/e2e_node/container_lifecycle_test.go

   Lines 902 to 903 in 7ad1eaa

   	err = results.RunTogetherLhsFirst(prefixedName(PreStopPrefix, regular1), prefixedName(LivenessPrefix, restartableInit1))
   	gomega.Expect(err).To(gomega.HaveOccurred())

I'm afraid I'm not sure what is expected to guarantee the liveness probe for the sidecar container (restartable-init-1) to run or stop before the prestop starts.

[hshiina]

Due to #124297 which was recently merged, another issue (#124704) appeared. Pod workers sometimes get blocked for a few seconds in kubelet like #124297 (comment). This may make something like race condition surface.

[SergeyKanzhelev]

/retitle [Flaking Test] [EventedPLEG] Containers Lifecycle should continue running liveness probes for restartable init containers and restart them while in preStop

[SergeyKanzhelev]

Marking with evented PLEG.

Is the issue also hapenning outside the evented PLEG?

[hshiina]

I don't think this happens outside the evented PLEG.
Usually, the init container gets into CrashLoopBackOff before the liveness probe for the regular container whose InitialDelaySeconds is 10 starts. So, the liveness probe for the init container does not run while the prestop is running.

If the pod worker works slowly with blocked by #124704, the init container may not get into CrashLoopBackOff.