Images should specify a path for a CA certificate bundle to be mounted by the runtime

[robbiemcmichael]

Motivation

Any software in a container that establishes a TLS connection requires CA certificates in order to perform the TLS handshake, these CA certificates are usually discovered from the filesystem. There is a longstanding practice of baking CA certificates into container images to simplify the process of running the container, however this presents several problems:

1. Reliability - Some images are updated very infrequently, which means CA certificates may be years out of date. This can result in an ecosystem of small time bombs that fail, not when a new version of the image is being deployed, but instead when a service the container connects to updates the certificate to one issued by a new CA which is not yet trusted.
2. Wasted effort - Companies with private CAs are forced to rebuild internal versions of many public images purely to add their own internal CA certificate to the bundle. This requires solutions which monitor new tags being pushed to the upstream image and trigger an internal build. The user experience is also poor as employees at these companies need to know where to find the internal version of the image instead of using the public image.
3. Security - Most people would have no idea what CA certificates are trusted in the images they use, which reduces the confidence they should have that they are establishing a connection with the expected service. I'm not aware of any container image scanning solution that checks that the CA certificates baked into an image are all well-known public root CAs.
4. Revocation - A compromised root CA would currently be a disaster for most teams running large numbers of containers, since removing trust for that CA would involve rebuilding every single image.

Proposal

The OCI image format specification should provide a standard way for images to specify a path for a CA certificate bundle, which can then be mounted by the container runtime.

Since there is no standard location for CA certificates, it naturally follows that it should be up to the image to specify the expected location for a CA certificate bundle.

This field should be optional and CA certificates will likely need to continue being baked into images for the time being to ensure that they are backwards compatible with runtimes that don't use this field. Eventually if runtimes widely adopt this feature, it may make sense to provide images with a way of registering a hard dependency on CA certificates being mounted by the runtime.

This proposal would require coordination with a similar proposal to the OCI runtime specification that the runtime should read this field and automatically mount a CA certificate bundle to this location. A sensible default would be to mount the host's CA certificates, but it also makes sense that runtimes may want to allow this to be overridden on a per container basis.

Benefits

This would help to solve some of the problems mentioned earlier:

1. Reliability - CA certificates would largely be kept up to date automatically. The process of updating the CA certificates on the host is simpler than updating them in every container that runs on the host. This would happen automatically in many cases, since most package managers automatically update the CA certificate bundle.
2. Wasted effort - Companies will not need to rebuild internal images purely to include their internal CA and users will not need to know where to find they company's internal images. Companies would only need to configure their runtime to use a bundle which includes their internal CA certificate. In most cases, the host will already trust the company's internal CA certificate so this will happen for free.
3. Security - The host is likely to be running a major Linux distribution. In my personal opinion, these distributions are usually a more trustworthy and up to date source for CA certificates than the average container image maintainer. The runtime should allow users to manually provide a CA certificate bundle to improve security further.
4. Revocation - Similar to the point above, if the CA certificate is removed from the bundle provided by the host's package manager, then this would happen automatically. If the runtime permits manual override then this would also provide an easier mechanism for people to revoke CAs even sooner, including revoking the company's own internal CA if required.

Further discussion

The implementation details are intentionally absent as I am seeking feedback on the concept before working on a more formal proposal for the changes to the image format and runtime specifications. Suggestions for how this could best be implemented are welcome however.

[jonjohnsonjr]

I've run into this myself. This seems like a good idea -- have you brought this up with the runtime-spec as well?

[robbiemcmichael]

@jonjohnsonjr I haven't brought it up there yet as I'm not sure of the preferred way to cross-post an issue which requires changes to both the image and runtime specifications. I could write up a short issue in the runtime-spec repo describing how the runtime might be able to make use of the CA certificate bundle path being stored in the manifest for the image, then I could link it to this issue for further context.

[loosebazooka]

This looks like a very nice idea. CA certs have been annoying for anyone trying to provide their own -- we've seen this in distroless and jib in the past -- but the solution has always been static with the aspiration of providing an at runtime solution. It's worth noting also that Java requires a jks file for certificates (which is why deb packages like ca-certificates-java exist), presumably the proposed format could accept multiple (ca-file,target) pairs?