Overhaul packaging, launching services within a omicron1 branded zones (#686)

This PR provides support for launching and managing services as zones.

### Nexus
- Exposes types in a way which can interoperate with Sled Agent.

### Sled Agent
- Parse a config file, rather than taking arguments over CLI
- Add a "Service Manager", responsible for zones that run independently of datasets (e.g., Nexus, Oximeter)

#### Networking
- Allow specifying static IP addresses for Zone addresses
- In doing so: Add some utilities to `sled-agent/src/illumos/zone.rs` to ensure that link-local addresses are available for IPv6 addressing. This lets us provide static IPv6 addresses for services, which we use elsewhere in the PR when setting up zones.

#### Storage
- Adds a "filesystem_put" endpoint to the Sled Agent, which allows a caller to request the creation of a local filesystem.
- Adds a "service_put" endpoint to the Sled Agent, which allows a caller to request the creation of a local service.
- Enable installation of `omicron1` branded zones.
- Overhaul the process of auto-loading installed zones and managing running zones.

#### Bootstrapping
- Enable the parsing of a `config-rss.toml` file, which provides requests that would *normally* arrive from the Rack Setup Service.
- Process the "rss requests" in the bootstrap agent, where they are injected before starting any local services.

### Packaging Tools
- Rely on [omicron-package](https://github.com/oxidecomputer/omicron-package) for package construction.
- Adds a new "zone" type package to omicron-package, allowing the creation of `omicron1`-branded `*.tar.gz` files.
- Adds a "cockroachdb", "clickhouse" packages to `package-manifest.toml`, with an associated `manifest.xml` file.

Author: smklein

.gitignore

@@ -6,6 +6,6 @@ logs
 out
 tools/clickhouse*
 tools/cockroach*
-clickhouse/
-cockroachdb/
+/clickhouse/
+/cockroachdb/
 smf/nexus/root.json

Cargo.lock

@@ -452,6 +452,14 @@ in a format which lets them be transferred to a Helios machine.
 This tool acts on a `package-manifest.toml` file which describes the packages to be
 bundled in the build.
 
+Configuration files are used to select IP addresses, and to manage Zpools
+utilized by the Sled Agent. These configuration files are located within
+`smf/`, and likely need to be modified to use addresses and zpools which match
+your hardware. Much of this configuration will be automated in the future
+(e.g., IP addresses will be inferred and posted to a DNS system, Zpools will
+automatically be detected on discovered disks), but for now it remains
+hard-coded.
+
 [source,text]
 ----
 $ cargo build
@@ -481,8 +489,10 @@ initialize other service as part of its setup sequence anyway.
 ----
 # List all services:
 $ svcs
+# View zones managed by Omicron (prefixed with "oxz_"):
+$ zoneadm list -cv
 # View logs for a service:
-$ tail -f $(svcs -L nexus)
+$ pfexec tail -f $(pfexec svcs -z oxz_nexus -L nexus)
 ----
 
 To uninstall all Omicron services from a machine, the following may be

README.adoc

@@ -108,3 +108,70 @@ pub struct InstanceRuntimeStateRequested {
     pub run_state: InstanceStateRequested,
     pub migration_params: Option<InstanceRuntimeStateMigrateParams>,
 }
+
+/// The type of a dataset, and an auxiliary information necessary
+/// to successfully launch a zone managing the associated data.
+#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq)]
+#[serde(tag = "type", rename_all = "snake_case")]
+pub enum DatasetKind {
+    CockroachDb {
+        /// The addresses of all nodes within the cluster.
+        all_addresses: Vec<SocketAddr>,
+    },
+    Crucible,
+    Clickhouse,
+}
+
+impl std::fmt::Display for DatasetKind {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        use DatasetKind::*;
+        let s = match self {
+            Crucible => "crucible",
+            CockroachDb { .. } => "cockroach",
+            Clickhouse => "clickhouse",
+        };
+        write!(f, "{}", s)
+    }
+}
+
+/// Used to request a new partition kind exists within a zpool.
+///
+/// Many partition types are associated with services that will be
+/// instantiated when the partition is detected.
+#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq)]
+pub struct DatasetEnsureBody {
+    // The name (and UUID) of the Zpool which we are inserting into.
+    pub zpool_uuid: Uuid,
+    // The type of the filesystem.
+    pub partition_kind: DatasetKind,
+    // The address on which the zone will listen for requests.
+    pub address: SocketAddr,
+    // NOTE: We could insert a UUID here, if we want that to be set by the
+    // caller explicitly? Currently, the lack of a UUID implies that
+    // "at most one partition type" exists within a zpool.
+    //
+    // It's unclear if this is actually necessary - making this change
+    // would also require the RSS to query existing datasets before
+    // requesting new ones (after all, we generally wouldn't want to
+    // create two CRDB datasets with different UUIDs on the same zpool).
+}
+
+#[derive(
+    Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq, Eq, Hash,
+)]
+pub struct ServiceRequest {
+    // The name of the service to be created.
+    pub name: String,
+    // The addresses on which the service should listen for requests.
+    pub addresses: Vec<SocketAddr>,
+}
+
+/// Used to request that the Sled initialize certain services on initialization.
+///
+/// This may be used to record that certain sleds are responsible for
+/// launching services which may not be associated with a partition, such
+/// as Nexus.
+#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema, PartialEq)]
+pub struct ServiceEnsureBody {
+    pub services: Vec<ServiceRequest>,
+}

common/src/api/internal/sled_agent.rs

@@ -28,7 +28,6 @@ pub mod api;
 pub mod backoff;
 pub mod cmd;
 pub mod config;
-pub mod packaging;
 
 #[macro_export]
 macro_rules! generate_logging_api {

common/src/lib.rs

@@ -64,6 +64,19 @@ impl From<types::InstanceState>
     }
 }
 
+impl From<omicron_common::api::internal::sled_agent::DatasetKind>
+    for types::DatasetKind
+{
+    fn from(d: omicron_common::api::internal::sled_agent::DatasetKind) -> Self {
+        use omicron_common::api::internal::sled_agent::DatasetKind::*;
+        match d {
+            CockroachDb { .. } => types::DatasetKind::Cockroach,
+            Crucible { .. } => types::DatasetKind::Crucible,
+            Clickhouse { .. } => types::DatasetKind::Clickhouse,
+        }
+    }
+}
+
 impl From<omicron_common::api::internal::nexus::InstanceRuntimeState>
     for types::InstanceRuntimeState
 {