Bootstore message handler implementation

[andrewjstone]

This is the second PR of the bootstore implementation. It creates handler methods for all existing messages and their corresponding DB requirements. RFD 238 section 5 will need to be updated to reflect this flow.

The way the flow currently is supposed to work is that there is a coordinator (coming in a follow up PR) that sends the following messages to initialize the trust quorum:

• Initialize
• KeyShareCommit (for epoch 0)

Initialize can be sent multiple times until the KeyShareCommit for epoch 0 locks in the configuration and Rack UUID. This allows the initial coordinator (RSS) to die, and even have its local storage or entire sled die, and still allow us to make progress via a new RSS. Importantly, once wired through the system, the coordinator will negotiate (via sled-agent and nexus) to write the KeyShareCommit to CockroachDB before sending it to all nodes. This ensures that the rack initialization will complete and CockroachDB only has to be setup once. There are some tricky failure modes here, but none of that is in this PR, so we can move on. The important point here is that once a KeyShareCommit for epoch 0 succeeds the rack is initialized.

After initialization, reconfiguration of the trust quorum can take place. Reconfiguration messages must be issued with the same Rack UUID. A reconfiguration consists of a KeySharePrepare for an epoch > 0 followed by a KeyShareCommit for the same epoch. After initialization a KeySharePrepare can never be mutated. However, to allow for certain failures, such as the failure of a node being prepared to ack (since we require unanimous acknowledgement in 2PC), we allow for new KeySharePrepares at higher epochs to proceed. We keep track of these epochs in CockroachDB to maintain linearizability. Once a node receives a prepare for a higher epoch, it will no longer accept prepare or commit messages for any lower epoch. Once a successful KeyShareCommit is handled, the sled is now operating at the new epoch.

GetShare requests are used to unlock trust quorum. They will rely on sprockets at the network level for authentication, confidentiality, integrity, and attestation. Some DeviceId membership validation will also be added to Nodein future commits as it currently exists in sled agent.

This is an incremental PR and there is still a lot to do here. Follow up commits will:

• Add more negative tests (possibly in this PR also)
• Add code for a coordinator
• Add a mechanism to store the encrypted (wrapped) root secret for each epoch and likely include it in each KeySharePrepare. This is the most straightforward way to transfer wrapped secrets. We can get more sophisticated in the future. Again, this needs to be written up in RFD 238, but I'm likely to just implement it within the next week and see how it works first.
• Add some key derivation from the root secret such that we can use these keys to encrypt and decrypt storage
• Add property based tests to simulate a full cluster along with failures and ensure rack unlock works according to the model and ensure that we can always unlock encrypted storage while tolerating our prescribed failure conditions within the model.

After that, the bootstore will basically be implemented at a non-network level. We'll then likely want to port over the sprockets code from the bootstrap agent to a higher level here, and use the Node and Coordinator from that level, which will end up interacting with Nexus and CockroachDB via the sled-agent. Multiple tasks will be utilizing the Node and its underlying Db at this point and so we'll need some concurrency control. I've given this a bit of thought and my current thinking is to have the Node running in it's own thread and serialize all operation via a channel. Outside of reconfiguration, all operations are reads, and we can simply cache the current share in memory for rack unlock purposes (e.g. GetShare). During reconfiguration, we'll mutate the DB as necessary, and refresh our cache. This strategy avoids the need for mutexes and has the characteristic that the Node and Db code we run in a single threaded manner in property based tests is actually running the same way in production with just a single DB connection. This strategy isn't set in stone, but in my mind it's the easiest to reason about and also the easiest way to hook async to sync code.

I tagged @jclulow here because he's a SQLite maven, and wanted a quick look to see if I'm doing things in a sane manner. Any tips would be appreciated. I expect @jgallagher to be my primary logic reviewer for the rest of this :)

[andrewjstone]

I know I wrote in the PR description that we should not accept a Commit if there is even a prepare for a later epoch. We don't do that here. I'm not actually sure if we should. For instance, is it possible that a node was prepared, offline for the commit, then somehow got a later prepare that included itself? That seems unlikely and maybe we can prevent it at the nexus layer. However, if we don't prevent it, we may still need the commit to allow the sled to unlock itself while a new prepare is in progress. I need to think about this some more.

[andrewjstone]

I"m leaning toward actually allowing this. I'm not sure it's harmful. I will continue to think about it as this protocol evolves.

[jgallagher]

LGTM 👍

[jgallagher]

Can epochs be negative?

[andrewjstone]

NOOOOOOOOOO. However, the type of INTEGER in sqlite is an i32, and even BIGINT is just i64. Rather than forcing lossy conversion I kept the types the same. I'm definitely open to changing this although it shouldn't be much of a problem here since we only start with epoch 0 and reject updates for earlier epochs.

[jgallagher]

I'm tempted to suggest making epoch a u32 in all the public APIs of this crate; the fact that SQLite forces us into i32 is bleeding out here. But if that's painful I don't think it's a huge deal given the checks.

[andrewjstone]

I'll think about it :)

[andrewjstone]

I'm not actually sure it's better to use a u32 given the constraints. I'm gonna leave it for now just to get this merged, but will keep it in mind.