K8SSAND-799 ⁃ Add support for service account token authentication to remote clusters

[jsanda]

In #82 I reported authentication issues with GKE. The OAuth token expires every hour and Google Cloud Tools is needed to renew it. We will face similar problems with other cloud providers. An alternative solution that is cloud provider-agnostic is to use service account tokens from the remote clusters. An approach was described in this article.

I propose that we create a script that does the following:

• Get the service account token from the k8ssandra-operator service account in the remote cluster
• Get the CA cert from the k8ssandra-operator service account in the remote cluster
• Create a kubeconfig for the remote cluster
• Create a kubeconfig secret in the control plane cluster
• Create a ClientConfig in the control plane cluster. It should reference the secret.

┆Issue is synchronized with this Jira Task by Unito
┆Epic: Multi-Cluster Deployment
┆Fix Versions: k8ssandra-operator-1.0.0-alpha.1
┆Issue Number: K8SSAND-799
┆Priority: Medium

[arianvp]

Note that in future Kubernetes versions static service account secrets will probably be phased out in place of ephemeral ones that are auto-renewed. Those service accounts will not have accompanying secrets anymore that don't expire. Instead service account token is made on demand and mounted directly into the pod. (And replaced when it expires)

https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md

kubernetes/enhancements#542

I think the plan is to enable this behaviour by default in 1.22.

This would mean you'd still run into expiring credential problems.

[jsanda]

Thanks for the references @arianvp. I need to read through those docs to better understand what is involved.

This would mean you'd still run into expiring credential problems.

As long as I can handle the expiration and renewal via k8s APIs then it should be fine. The expiration problem I ran into required either have gcloud installed in the operator image or making calls to GCP APIs. We would likely have to do something similar for each cloud provider.

[jsanda]

The KEP mentions this about expiration:

A volume plugin implemented in the kubelet will project a service account token sourced from the TokenRequest API into volumes created from ProjectedVolumeSources. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.

It looks like there is a mechanism to handle expiration, but this wouldn't apply to our use case of using the token out of cluster. We will need to sync the tokens, but I don't know that using the k8s api is the best way to do that. If for some reason the token does expire before it can be synced then we might be stuck in a bad state.