Consider supporting Tang offline provisioning

[jlebon]

Feature Request

Environment

Applicable to any environment where disk encryption is desirable.

Desired Feature

I was recently made aware that Tang supports offline provisioning by obtaining the advertisement out of band and passing it directly to Clevis (see the final paragraphs of this section).

This could be useful for example to provision a machine in one environment, and then move it into its final location where a secure connection to the Tang server is available for subsequent boots. (Edit: actually the use case that motivated this RFE is described in #1474 (comment).)

We should consider adding support for this in Ignition.

Other Information

Using a custom pin for this almost works:

"clevis": {
  "custom": {
    "config": "{\"url\": \"http://192.168.122.1:8000\",\"adv\": {\"payload\": \"...\",\"protected\":\"...\",\"signature\":\"...\"}}",
    "needsNetwork": true,
    "pin": "tang"
  }
},

There are two problems however:

1. needsNetwork controls networking for both first boot and subsequent boots, but we don't actually need the network on first boot and one may not even be available.
2. the disks stage by default will close and reopen the devices as a sanity-check; re-opening will fail if Clevis can't reach the Tang server.

[jlebon]

Not requiring networking on first boot for Tang encryption could also provide some breathing room in some situations wrt network configuration.

For FCOS/RHCOS, the Tang UX issue is related but I don't think it significantly helps it.

[jlebon]

More details on the use case for this. It's not so much about provisioning in a separate environment, but rather about making provisioning more stable with you have multiple Tang servers.

Imagine you have e.g. 5 Tang servers for availability (threshold 1). Currently, all 5 servers need to be online at provisioning time in order for Clevis to bind the LUKS device. With offline mode, we no longer need to contact the Tang servers on first boot. And for subsequent boots, unlocks will respect the threshold of 1. So at no point will the process require all 5 servers to be online.

[bgilbert]

I think the right way to support this is by adding a config spec field, rather than teaching Ignition to parse the custom Clevis pin.

[bgilbert]

Specifically, I think this should work:

• clevis (object): describes the clevis configuration for the luks device.
  • tang (list of objects): describes a tang server. Every server must have a unique url.
    • url (string): url of the tang server.
    • thumbprint (string): thumbprint of a trusted signing key.
    • advertisement (string): pre-shared server advertisement JSON.

I think the advertisement is best treated as an opaque blob, rather than trying to parse out its fields. There's already precedent for embedding opaque JSON as a string (the clevis.custom.config field).

If the advertisement field is present, we can then avoid requiring network as a result of this particular Tang server, and can also bypass the sanity check. The latter isn't great, but I don't see a way around it for this use case.

[bgilbert]

• It seems reasonable for users to provide a Tang advertisement for some servers but not others. For example, there might be a server in the same rack, where the user doesn't bother providing an advertisement, and a backup server elsewhere, where the user provides an advertisement because the server is more likely to be unreachable.

• I think we should continue performing the close+open check whenever we can, since its purpose is to prevent data loss due to accidentally creating a volume that can't be unlocked after the machine is rebooted (some weeks later). We can still perform the check if n_tang_servers_without_advertisement_specified + int(use_tpm2) >= threshold, since a successful clevis bind implies that we have enough key shares to reopen the volume. But we could also go further, and perform the check if own server probes (which we already have some code to do) show that enough servers are accessible, even if the above test wouldn't enable the check.

• I'd also argue for adding a second cross-check: if an advertisement is provided, and the server is up, we should check that the server offers the same advertisement. Clevis doesn't do this, but there's value in flagging the fact that e.g. the Tang keys were rotated but the Ignition config hasn't been updated. There are theoretically cases where the user might intentionally configure a mismatch (e.g. we're provisioning a link-local Tang server but the first boot is on a different link), but I think we should start restrictive and see if it breaks anyone.

cc @jlebon for any thoughts

[jlebon]

We can still perform the check if n_tang_servers_without_advertisement_specified + int(use_tpm2) >= threshold, since a successful clevis bind implies that we have enough key shares to reopen the volume. But we could also go further, and perform the check if own server probes (which we already have some code to do) show that enough servers are accessible, even if the above test wouldn't enable the check.

Opportunistically doing the reopen check seems reasonable, though the non-determinism introduced by probing is a bit awkward. And hmm, I guess this races with network startup too? The probing won't be able to retry forever because it's optional. And networking might come up after we've finished retrying.

[bgilbert]

Oh, good point about racing with network startup. Do you think it's better not to check at all than to do it best-effort?

[jlebon]

Oh, good point about racing with network startup. Do you think it's better not to check at all than to do it best-effort?

I'm not sure. I think opportunistically doing something seems a bit uncharacteristic of Ignition's otherwise deterministic stance so far. I also think users wanting to use the offline provisioning feature might get confused seeing Ignition still trying to reach the Tang servers over the network. So overall, I'm inclined to just skip it for now. But it's not a strongly-held opinion.

[bgilbert]

Yeah, that's fair. Discussed this with @prestist OOB and we agreed that the nondeterminism of the extra checks could be surprising to users. So in #1474 (comment), we'll drop the third bullet and the second half of the second bullet.