403 unauthorized - BoundServiceAccountToken and token refresh support. #675
Description
Kubernetes version 1.21 defaults to enabling the beta version of BoundServiceAccountTokenVolume by default. Service account tokens now have an expiration of one hour, which means that clients that rely on these tokens must refresh the tokens within an hour.
We've run into keel getting 403 API errors after the tweaked 90d refresh interval has expired on EKS. This was fixed by a Pod restart, thus extending the period to another 90d, but this is a hack that will become bothersome for those environments that use the default 1h period.
It looks like as if the Kubernetes client SDK for Go automatically refresh tokens within the required time frame, so supporting newer k8s versions might be a matter of upgrading the SDK.
Sample trace of the error once the refresh interval is up:
1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.1-0.20190516230509-ae8359b20417+incompatible/tools/cache/reflector.go:94: Failed to list *v1.DaemonSet: Unauthorized