KEP-3257: cluster trust bundles: add a new in-tree signer and its CTB

[stlaz]

• One-line PR description:
  Adds a mention of a new default trust bundle and its signer + one more non-goal

• Issue link: ClusterTrustBundles (previously Trust Anchor Sets) #3257

• Other comments:

/cc ahmedtd enj liggitt

[liggitt]

A good exercise to make sure we fully specify the meaning / behavior of this signer is to add the documentation here for this new signer that we would add to https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers.

Because this is encompassing existing serving certificates issued by other existing signers, we have less control over the exact shape, but we can say some things like it must be valid for in order to work, e.g.:

• Permitted x509 extensions - honors key usage and DNSName/IPAddress subjectAltName extensions. At least one DNS or IP subjectAltName must be present.
  • Conformance test requires resolution of kubernetes.default.svc.$cluster-domain - I'm not sure the current status of the ability to change the cluster domain from cluster.local ... maybe get network folks feedback on whether it makes sense to spec that the kube-apiserver certificate should (must?) contain the DNS subjectAltName kubernetes.default.svc and/or kubernetes.default.svc.$cluster-domain
• Required key usages (other key usages may be present) - ["key encipherment", "digital signature", "server auth"] or ["digital signature", "server auth"].

This line in the signerName docs would also need to change if we don't intend to have k-c-m sign these certs

The kube-controller-manager implements control plane signing for each of the built in signers.

[stlaz]

Because this is encompassing existing serving certificates issued by other existing signers

It's encompassing certificates you can use to verify kube-apiserver serving certificates, but it's not restricted to bundling the serving certs, right?

If I were to maintain a cluster, I wouldn't actually want to put a serving certificate in this bundle. Instead, I want to bundle signers of those serving certs so that I can rotate my KAS serving certs freely, without having to worry about what's going to happen to the KAS trust in all the pods in the cluster.

I don't think we can easily restrict the certs in the bundle, except for we could probably ban the use of the "client auth" EKU. I think we could force the restrictions you described on certs with the server auth EKU.

Note that your proposal would likely make the bundle unusable in OpenShift, if I understood it correctly.

[stlaz]

I added a commit with a section that talks of validation of this bundle, please see if it makes sense to you. I haven't run the DNS/IP SAN part through the sig-networking folk yet.

[enj]

It's encompassing certificates you can use to verify kube-apiserver serving certificates, but it's not restricted to bundling the serving certs, right?

I think Jordan is saying this new signer takes ownership of the existing API server serving certs. So for example, we cannot say something like "the SAN list but be exactly kubernetes.default.svc and kubernetes.default.svc.$cluster-domain" because existing working clusters could have extra SANs in their serving certs. I understand Jordan's comment to only be about the signer, and not the CTB contents.

[stlaz]

yes, this was clarified in a later comment - #4791 (comment)

[liggitt]

the Upgrade / Downgrade Strategy and Version Skew Strategy need attention to describe:

• default enablement order between the beta API and the pod field
• behavior of skewed kubelets when the beta API is unavailable but pods from a newer API server try to mount the cluster trust bundle
• behavior of skewed kubelets when the feature gate is disabled but pods from a newer API server try to mount the cluster trust bundle

(it's ok to make those changes in another KEP update PR, just noting that they're needed)

[stlaz]

Many thanks for the reviews! I'm still to work on the Upgrade / Downgrade Strategy and Version Skew Strategy parts.

I'm not sure I understood the part about the new bundle certs' restrictions properly. I think that having some restrictions would be a good thing to help users avoid putting wrong certs into the bundle, let's clarify some more how we'd like to handle those.

[stlaz]

Added explanation on version skewed kubelets. Also realized the KCM behavior should probably also be featuregated so I added a FG for it.

[liggitt]

I think these rules are mixing requirements for the certs in the bundle, and certs issues by signers in the bundle. Some of these rules are for the certificates issued by signers in this bundle, not the CA certs in the bundle themselves, right?

I'm mostly looking for the rules on the kube-apiserver certs (serving certs, with some guidance on expected SANs)

[stlaz]

I'm mostly looking for the rules on the kube-apiserver certs (serving certs, with some guidance on expected SANs)

I see, that's the part I wasn't sure about. It's a bit odd situation with the signer because we won't be the ones doing the signing this time. Enforcing the rules gets a bit tricky this way, although a conformance test is a good idea on how to make sure our expectations on the signer are met.

I'll think about this section some more.

[stlaz]

I rewrote the whole section, it should now match the format of the signers from the k/website. I made the requirements on the signed certs' SANs a bit looser to allow for non-internal kube-apiserver hostnames, I wonder if that's something we may want to allow?

[enj]

First pass on the diff. I think there is still stuff in this KEP that needs to be changed such as removing

Signer controllers may follow the convention of making the label selector kubernetes.io/cluster-trust-bundle-version=live correspond to a meaningful set of trust anchors.

[enj]

It's encompassing certificates you can use to verify kube-apiserver serving certificates, but it's not restricted to bundling the serving certs, right?

I think Jordan is saying this new signer takes ownership of the existing API server serving certs. So for example, we cannot say something like "the SAN list but be exactly kubernetes.default.svc and kubernetes.default.svc.$cluster-domain" because existing working clusters could have extra SANs in their serving certs. I understand Jordan's comment to only be about the signer, and not the CTB contents.

[stlaz]

Rewrote the API discovery for KCM, kubelet to be only performed upon startup.

[stlaz]

The latest push only squashes most the commits.

[enj]

/lgtm
/approve

(Jordan said he was fine with my ack)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enj, stlaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-auth/OWNERS [enj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment