[BUG] Intermittent timeouts due to apiserver shutdown

[jemag]

Describe the bug
Workloads in the cluster will experience intermittent timeouts to the API server. When we investigate and correlate these API server timeouts, we notice that they coincide almost always with the apiserver shutting down.

For example,
Here we can see our KEDA workload timing out at around 21:45:17:

and here are the logs within the API server:

Similar example for cert-manager:

and api-server shutdown:

Another example with kyverno:

and api-server shutdown:

To Reproduce
Steps to reproduce the behavior:

1. Run workloads that require connection to the API server
2. Notice intermittent timeouts and correlate with api-server logs. Notice api-server shutdown

Expected behavior
There should not be a timeout whenever the api-server shuts down unless extraordinary circumstances.

We do not have vision on how the API-server is setup on AKS side, but we would expect proper behavior such as:

• There are multiple containers answering requests to the API server
• If one of the container needs to shutdown, it should
  • Stop accepting new requests
  • do a proper graceful terminating where remaining requests are fully answered before fully shutting down

Additional Context:
sku_tier is Standard

Environment (please complete the following information):

• Kubernetes version v1.30.5

[microsoft-github-policy-service]

Action required from @aritraghosh, @julia-yin, @AllenWen-at-Azure

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads