NodeNotReady test flakes on Release-1.3 test jobs

[nawazkh]

Which jobs are flaking?

• Spotted in periodic-cluster-api-e2e-release-1-3.
  • One of the instances of such a failure periodic-cluster-api-e2e-release-1-3/1698294486968307712

Which tests are flaking?

Not really applicable since the test never gets triggered.

Since when has it been flaking?

The exact start date is unknown, but this flake seems to be evident since Sept 1st.

• Around Sept 1st

We migrated all nodes in the EKS Prow build cluster to a public subnet, so all nodes have public IP addresses instead of routing all traffic via a NAT Gateway.

Refer to: kubernetes/org#4433 (comment)
Refer to : https://kubernetes.slack.com/archives/C8TSNPY4T/p1694020825316969

Testgrid link

https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.3#capi-e2e-release-1-3

Reason for failure (if possible)

The test does not get triggered. The pod info, upon scrolling all the way down says Node not ready

Anything else we need to know?

This might be related to the thread going on in #sig-k8s-infra on Nodes are randomly freezing and failing 🧵

Label(s) to be applied

/kind flake
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[sbueringer]

/triage accepted

[sbueringer]

Would be good to link the Slack thread directly :)

[sbueringer]

Did we talk to sig-k8s-infra and ask them if our problems could be related to their problem?

[nawazkh]

Would be good to link the Slack thread directly :)

Absolutely! I wasn't sure if it was ok to link a Slack discussion.

[nawazkh]

Did we talk to sig-k8s-infra and ask them if our problems could be related to their problem?

Thanks for bringing this up. I was not sure if we wanted to triage this on our end first.
But I posted a message on their thread to get it to their attention. https://kubernetes.slack.com/archives/CCK68P2Q2/p1694108045460259?thread_ts=1693476605.123389&cid=CCK68P2Q2

[sbueringer]

Good point, sorry I forgot what I said yesterday :). But I guess we're probably already at the point where we can't do much

[abdurrehman107]

Another few jobs I spotted failing were:

• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-release-1-3/1699382925251842048
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-workload-upgrade-1-22-1-23-release-1-3/1699451358442164224
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-workload-upgrade-1-25-1-26-release-1-3/1699451107115274240

[sbueringer]

Any news? (I was on PTO the last few weeks)

[nawazkh]

I am following up on #sigs-k8s-infra over here https://kubernetes.slack.com/archives/CCK68P2Q2/p1698254656470629?thread_ts=1694458280.112599&cid=CCK68P2Q2

Thanks for the reminder!

[sbueringer]

@ameukam Mentioned during KubeCon that they worked around the issue by using a different OS for the Nodes of the Prow cluster (IIRC). @nawazkh Any new occurences of this issue?

[sbueringer]

Hm @ameukam as far as I can tell the issue still exists: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-mink8s-release-1-3/1724739536589688832

[chrischdi]

FYI, still getting jobs with Pod got deleted unexpectedly from time to time.

Example: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-mink8s-main/1736768378699255808

[sbueringer]

@chrischdi can we open a new issue and close this one? The title is highly misleading

[chrischdi]

Created follow-up issue #9901

/close

Because release-1.3 jobs are gone.

[k8s-ci-robot]

@chrischdi: Closing this issue.

In response to this:

Created follow-up issue #9901

/close

Because release-1.3 jobs are gone.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.