Start implementing bootstore: Replicated storage required for rack unlock

[andrewjstone]

Currently RSS is in charge of generating the rack secret and the bootstrap agent is in charge of establishing sprockets sessions and transferring shares required for rack unlock. The shares themselves, and the trust quorum membership is stored in a file on each sled. The major problems with this implementation are that it is not resilient to failure and does not allow reconfiguration of the trust quorum. Additionally, the trust quorum code is somewhat scattered across the sled-agent, and a little hard to follow.

This change introduces the bootstore, which will become responsible for all trust quorum protocol and storage on the rack. The bootstore will be instantiated by the boot agent, which will be responsible for establishing sprockets sessions that the trust quorum can operate over. This allows for a readable, testable implementation of the protocol and storage without having to worry about extraneous details related to networking or the SP. We may choose to pull that in later to a higher level module in the bootstore. I intend to utilize property based tests with fault injection to cover the protocol.

This is an early WIP of the bootstore, and lays out the basic structure of the code, while implementing some small bits. The goal is to get this out early before it gets too large to easily review.

The Node is the main entry point to the codebase, although most logic has not been implemented. The Node uses the Db to query and store trust quorum information. The goal is to build up a fairly general two-phase commit mechanism that can also be used for storing networking information required to setup NTP so that CockroachDb can boot and the rest of the control plane software can run. The two-phase commit is standard, but as always has the problem of any node failing halts the protocol. We will manage this by moving to a new epoch and reconfiguration if this occurs. Of course the devil is in the details, but it's not incredibly complicated. This implementation will supercede what is written in section 5 of RFD 238. I will update that document with the new protocol. One thing that was not considered there was transferring the encrypted (wrapped) root secret to new nodes that don't have the original root secret derived from the epoch 0 rack secret. As part of this, we'll distribute it as part of the 2-phase commit. There are a couple mechanisms with various levels of security, that I won't get into here. The important thing to know for this PR is that I created a table and model for storing the secret. Code to use it is unimplemented so far.

There is also the notion of a [Coordinator] that is not yet implemented. The coordinator interacts with the sled-agent to record important information in nexus and get instructions related to trust quorum from nexus. It only stores state in memory and can crash without consequence. It may end up being driven by a saga.

I also copied the trust_quorum directory from the sled-agent as the bootstore will become responsible for the trust quorum. I haven't yet removed this code from the sled-agent, as this work is preliminary and I don't want to hook it in yet.

[andrewjstone]

@jgallagher One more thing I should mention with regards to this PR. There's no locking or concurrency support of any kind on the DB connection yet. That's because I haven't figured out what I want to do. I want to get more code written first. In the majority of cases, the DB will be Read-Only, and I'm not sure if we want to just cache state at the app layer and use a single connection with an Arc<Mutex<SqliteConnection>>, do this without the app layer caching, use multiple connections, possibly one per Node created on demand, a connection pool, or something else. In short, don't worry about concurrency on this particular PR. It'll get decided upon and implemented shortly in a follow up.

[jgallagher]

LGTM; I appreciate splitting this out into an initial PR before it becomes larger - thanks!

[davepacheco]

I wanted to add some notes on the pq-sys stuff since we spent some time digging into it. To summarize:

• we have a new crate
• it wants a dev dependency on omicron-common
• it thus depends on pq-sys (via diesel)
• it must thus use the omicron-rpaths pattern, which involves a build dependency on omicron-rpaths and a regular dependency on pq-sys version "*"

We observed this behavior:

• if we have a dev dependency on pq-sys, and if the build.rs runs unconditionally, then you get a compile-time error about DEP_PQ_LIBDIRS not being set. This happens because this variable is set by Cargo when that variable is provided by a non-dev dependency that emits the variable from its build script. pq-sys emits this. But since we're using it as a dev dependency in this case, it's not there in our build script.
• if you have a dev dependency on pq-sys, and if the build.rs is wrapped with #[cfg(test)], then you get a runtime error from ld.so.1 about not finding libpq. You might think this solution would do the right thing by causing non-test binaries to get no RPATH entries (since the build script wouldn't have done anything) and test binaries to get the right RPATH entries (since the build script would use omicron-rpaths in that case). But we concluded that build scripts are run only once for the whole package, not once per crate (binary) built by the package. It presumably gets run without #[cfg(test)]. So you just wind up building a binary that depends on pq-sys (because it's a dev-dependency and you're building a unit test) but that's missing the RPATH entry. Hence this failure mode.
• If we have a regular dependency on pq-sys, it all works.

Because build scripts appear to only run once and also only get metadata from regular deps (not dev deps), the conclusion is that if your package ever depends on pq-sys directly or indirectly, even via dev-dependency, then you must apply the pattern by adding a regular dependency on pq-sys. You can't use a dev-dependency for pq-sys, even if you only depend on pq-sys via a dev-dependency.