Consider removing direct ACME support for bundle endpoints

[azdagron]

Endpoints which return bundle information must be protected by TLS to secure the transmission of these security critical pieces of information. When implementing both the SPIRE server federation bundle endpoint and the OIDC discovery provider JWKS endpoint, the decision was made to incorporate ACME support as a way to facilitate easy TLS protected deployment when SPIFFE authentication itself was not desired.

Building in ACME support has led to some less than ideal interactions, particularly in SPIRE server. First, in order to simplify key management we decided to fork the popular golang.org/x/crypto/acme/autocert package to support the key manager plugin. This was not a straightforward fork and keeping up with it has been meddlesome. Further, KeyManager operations can be expensive, tying the TLS handshake for the bundle endpoint to the KeyManager can have unseen costs since can often be an internet-facing endpoint. Additionally, our implementation doesn't even properly manage the keys during key rotation, leading to a period of broken TLS handshakes during ACME renewal (#2167).

While we want to make it easy to do the right thing, building in ACME support was probably a step too far. Many proxies have built in ACME support and would be trivial to stick in front of SPIRE or the OIDC discovery provider to provide TLS protected access.

In order to make it hard to do the wrong thing, we might consider only making the bundle endpoint available over UDS when it isn't being protected with SPIFFE authenticated TLS. This makes it hard for somebody to expose a plain text TCP port, or at least makes it an explicit action by an operator.

[azdagron]

#2032, particularly the last bullet point around supported the whole gamette (including future introductions) of TLS CipherSuites makes me further question the use of the KeyManager with ACME. Even if we end up keeping ACME support, I wonder if we should move away from KeyManager managed keys for ACME. The implications there are of course that either we only support in-memory, which could cause a large number of renewals if a server was broken and being restarted over and over, or if we'd go back to on-disk key support, which is an unfortunate position that we don't like to be in.

[azdagron]

Discussed in contributor sync. We'd still like to do this. The work needs to be scoped a little more. Open questions:

1. Do we remove ACME support in its entirety?
2. What do we recommend instead (creds on disk? insecure locally bounded port behind nginx or another proxy that does tls termination)?
3. Do we remove the SPIRE server bundle endpoint server completely in lieu of bundle publishers (e.g. bundle published to s3)?

[kfox1111]

For the helm-charts-hardened chart, we found the acme support was rather problematic in the discovery provider. We switched it to use cert-manager for the acme support. The Kubernetes support needs to be able to persist the state beyond the pod lifecycle, so doing it with cert-manager allows that state to persist in k8s secrets rather then pvcs and handle multiple instances better.

For the standalone use case, some kind of acme support may still make sense, but maybe just supporting existing cert/key is enough and something like cert-bot can handle the acme part reliably and more flexibly.