Support sign/verify with X.509 certificates

[donpellegrino]

Description

Singularity currently supports PGP/GPG web of trust for digital signatures. It would be useful to also support hierarchical chain of trust with X.509 certificates. This would provide advantages for users and system administrators having Certificate Authority (CA) certificates already in place in the OS, since they could trust signed containers without additional steps for verification of the signer.

Desired Solution
The ability to sign containers with X.509 certificates. The ability to verify containers against CA certificates already in place in the host OS.

Describe alternatives you've considered
PGP/GPG key management servers can provide another mechanism for trusting certificates. However, some organizations already have established CA cert. processes and lack PGP/GPG key management servers. It would be useful to support the organizations existing CA certs.

Additional context
Certificate providers such as IdenTrust provide code signing certificates. It would be useful to be consistent with the processes used for code signing and verification established by such providers and implemented in other software execution environments.

[DrDaveD]

FYI there is currently a pull request to do this pending on the Apptainer project.

[tri-adam]

@donpellegrino, thank you for the request. Makes sense to me.

@DrDaveD, thanks for the pointer. That PR appears that depends on code not present in the integrity package used by Singularity. Not necessarily a blocker, but noting that here since it's relevant to the scope of work.

[DrDaveD]

Right, that came in another PR and a followup PR in the apptainer fork of sif.

[tri-adam]

Some notes on progress here:

• An on-disk format (DSSE) has been proposed for this work in a CE Slack thread and a CE Google Groups thread. Thanks to @fnikolai and @jscook2345 for contributing to the discussion.
• Support for DSSE encoding/decoding was merged in SIF: DSSE support sif#228
• A public API for non-PGP signatures has been proposed in SIF: Expose non-PGP digital signature support sif#245
• A draft PR is up with user-facing support for basic sign/verify using non-PGP public/private keys: Sign/Verify with Non-PGP Key Material #1120

[tri-adam]

Going to take a stab at the UX design here for non-PGP sign/verify in general, of which X.509 verification is a subset. Would love any feedback you might have @donpellegrino @fnikolai @DrDaveD @dtrudg

Sign

The non-PGP sign flow is straightforward, as proposed in #1120. We provide a PEM-encoded private key, and it is used to generate the signature:

singularity sign --key private.pem image.sif

Verify

The non-PGP verify flow requires public key material, but there are different ways this can be provided.

Verify with Public Key

The simple case is to verify directly with public key material, as proposed in #1120. In this case, all we need to do is provide a PEM-encoded public key:

singularity verify --key public.pem image.sif

Verification should pass if the public key material in public.pem matches the private key material in private.pem.

Verify with Certificate

The more complicated case is to verify with an X.509 certificate. In this case, we provide a PEM-encoded certificate that contains public key material:

singularity verify --certificate cert.pem image.sif

Verification should pass if the certificate in cert.pem is valid, and contains public key material that matches the private key material in private.pem. Of course, what constitutes a valid certificate may vary depending on the environment and/or use case. It likely includes some of the following:

1. The certificate isn't expired.
2. The relevant public key material within the certificate is marked as being valid for code signing.
3. The certificate has been signed directly by a root CA, possibly via a chain of one or more intermediate CAs. This likely involves some options:
   1. An optional pool of intermediate certificates (ex. --certificate-intermediates). Intermediates is an optional pool of certificates that are not trust anchors, but can be used to form a chain from the leaf certificate to a root certificate.
   2. An optional pool of trusted root certificates via --certificate-roots, which the certificate needs to chain up to. If not supplied, the system roots are used.
4. The certificate(s) involved have not been revoked, according to online revocation checks (OCSP) and/or a certificate revocation list (CRL). Neither, one of, or both of these could theoretically be relevant for a given environment, so I'd suggest we address this via options:
   1. The user can optionally specify --ocsp-verify, in which case online revocation checks are performed. We would need to decide whether this should be strict or permissive in the event that an OCSP server is unavailable.
   2. The user can optionally specify CRLs, some how. I haven't actually heard the CRL use case come up, so perhaps we put that to the side for now?

The first three points seem like a bare minimum to have a useful X.509 capability. @donpellegrino and @fnikolai (and anyone else following this thread), could you comment on revocation check preferences/requirements in your environment? Are there any other requirements we can identify up front?

[fnikolai]

@tri-adam Seems solid. My only "UX concern" is that verify command will become bloated by the numerous flags.

I understand its "backward-compatibility" merits, but "--certificate", "--ocsp-verify", etc, are irrelevant to PGP verification.

One solution will be to panic/fail upon invalid combinations of flags to raise the user's attention.

Another solution will be to create a "subcommand" that will confine its configuration flags (e.g, verify x509 --certificate...)

I tend to prefer the second approach, as it's more "future-proof".

[dtrudg]

@tri-adam - 1 through 3 seem sensible to me. I think talking through 4 to far should be postponed for further input from hopefully multiple potential users.

@fnikolai - I'm sympathetic to the concern r.e. many flags on a command. However, I'd argue that adding a subcommand is going to cause more friction for users. If we have system / sysadmin managed certificate pools etc. in future, then it should be possible to do a singularity verify without needing to know whether the image is PGP or x509 signed ahead of time. It's difficult to step backward from a subcommand to this.

[fnikolai]

@dtrudg I see. Perhaps something along these lines?

$ singularity sign --help
...
PGP flags:
      --whatever string                                                                                                                                                                                                      
                Some Whatever message Here
      ....

X509 flags:
      --certificate
                Path to x509 certificate
      --key
                Path to x509 key
      ....

There is something called 'NamedFlagSets' that could help us with grouping the flags.
spf13/cobra#1327

[dtrudg]

@fnikolai - yes, laying it out like that would be great, when we can. Singularity does some weird stuff with cobra flags, but we can follow that issue to see how it progresses.

[tri-adam]

Thank you for all the feedback!

With regard to separate verify commands, I'll throw in another reason it might make sense to have a single command... the user doesn't necessarily know how an image was signed when they run verify. Since we can auto-detect this, at a minimum we can give useful feedback when, for example, non-PGP key material is required to verify a given image.

I put a minimal implementation up in #1139. By minimal, I mean it address only points one through three in the list above, and no revocation checking is performed. The code is structured in a way that should make adding optional revocation checking straightforward.

@fnikolai I haven't separated the options out yet as you suggested. The sign command isn't actually substantially expanded in options. verify certainly does suffer from a growing list of options, which might warrant further organization as you suggest.

Lastly, @donpellegrino, if you have any feedback on whether your use case would require online (or even offline) revocation checking, it would be good to get that out here. Thank you!

[donpellegrino]

Lastly, @donpellegrino, if you have any feedback on whether your use case would require online (or even offline) revocation checking, it would be good to get that out here. Thank you!

For verification and revocation checks, I would leave the management of host certificates in the chain-of-trust out of scope. Managing the host certificates would be left to the maintainer of the OS. I could imagine where an online check of a certification revocation list would be convenient, but I would make that optional and not treat lack of connectivity to an online list as an error. The online check might be an optional, non-default feature. I am thinking of HPC and/or air-gapped use cases where hundreds or thousands of containers were running on separate nodes. It would be undesirable for any default behavior to trigger network traffic. Ideally, each Singularity instance would default to use the locally available information exclusively and not initiate any network traffic.

[tri-adam]

Sounds like everyone is singing the same song, roughly. Online verification checks are useful, but should be optional.

From the sounds of it @donpellegrino, the above scoped work is a superset of your base requirements, since it includes the ability to (a) optionally utilize a pool of root certificates other than the system root, and (b) optionally specify a pool of intermediate certificates that can form a chain between the leaf certificate and the root pool.

Thanks so much for circling back! Looking forward to getting the base functionality merged.