Sled Agent must manage durable storage for configs, zones, explicitly

[smklein]

See also: RFD 118

Sled Agent currently configures a few pieces of data outside datasets explicitly allocated in pools:

• /var/oxide contains a variety of configuration information, including...
  • RSS setup information
  • The "Sled Request" used to launch the sled-agent (including underlay information)
  • A list of "All services which should be launched on this sled"
  • A list of "All services which should be launched on u.2 zpools"
• /zone contains the "filesystem for all zones"
• /opt/oxide contains all the "latest installed system images", and is used to update control plane software which exists outside the ramdisk.

Q: So, why is this bad?
A: All those paths are currently backed by a ramdisk -- specifically rpool -- on gimlets.

This means that when we reboot, a significant portion of the necessary configuration information to launch the sled will be lost. Furthermore, for the zonepath filesystems, a significant portion of user RAM will be dedicated to zone-based filesystems, which we'd prefer to distribute to disk-backed file storage.

Here's a list of some of the work we need to accomplish to mitigate this in a production environment:

• Sled agent should be responsible for formatting U.2s and M.2s with necessary datasets
  • See: [sled agent] Should be capable of parsing, formatting M.2 drives #2688
• Sled agent should move "sled-agent-request" information into the M.2 partitions, duplicating them on write. #2998
• Sled agent should move all RSS-related information into the M.2 partitions, duplicating them on write. #2970
• Sled agent should move all service-related information into the M.2 partitions, duplicating them on write. #2969
  • This would be made easier with Tracking issue for "Self-assembling Zones" #1898 , as we'd only need to store the zone information, and not the corresponding config for "how to launch the zone".
• Sled agent should allocate zonepaths from U.2s
  • This means sled agent needs to handle the allocation / de-allocation of zonepaths much more explicitly.
  • Related: [sled agent] Creating zones for Propolis Instances leaks datasets #1119 , Crucible datasets remain after disks are deleted. #1313
• Sled agent should move all software images from /opt/oxide into /pool/int/<UUID>/install. #2971

[andrewjstone]

I just remembered that #3007 adds some support for persisting zone filesystems under crypt/zone.

This code has been ready to go for a few weeks now and has been tested numerous times. I've done cold boot testing on paris with it, but since there is no persistent install of software on the M.2s like dogfood, I have to re-run omicron-package install after reboot to rediscover and mount the U.2 encrypted datasets. The PR has the details about this. While I'd like to test with a proper install on dogfood I"m at about the point where I'd like to just merge this and see what happens.