Failing e2e: restart kube-controller-manager fails

[msau42]

Is this a BUG REPORT or FEATURE REQUEST?:
@kubernetes/sig-storage-bugs
@kubernetes/sig-auth-bugs

What happened:
Since 8/31 10am, the following test case fails. It kills kube-controller-manager and waits for it to restart. However, it seems like kube-controller-manager failed to come back up:
[sig-storage] NFSPersistentVolumes[Disruptive][Flaky] when kube-controller-manager restarts should delete a bound PVC from a clientPod, restart the kube-control-manager, and ensure the kube-controller-manager does not crash

Testgrid
Logs

kube-controller-manager failed to come up with this error:

I0904 17:53:07.775832       1 serving.go:293] Generated self-signed cert (/var/run/kubernetes/kube-controller-manager.crt, /var/run/kubernetes/kube-controller-manager.key)
failed to get delegated authentication kubeconfig: failed to get delegated authentication kubeconfig: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

[liggitt]

likely #64149

/assign @sttts

[msau42]

The other test cases that test kubelet restart also become very flaky at the same time. I suspect those tests will fail setup if they are run after this kube-controller restart test case

[sttts]

It looks like the kube-controller-manager is started as static pod in the gci e2e test. When it first starts, there is no kubernetes service and the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT env vars are not set inside the container.

When the kube-controller-manager is restarted, the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT env vars are set, but the static pod has no service account, i.e. /var/run/secrets/kubernetes.io/serviceaccount/token does not exist. We made the later fatal in rest.InClusterConfig and its use to setup delegated authn/z.