Sled Agent should be capable of utilizing swap space on the M.2 dump device

[smklein]

Parent bug: #2688

[jclulow]

NB: I have some issues at a design level I've outlined in a comment on the PR: #2859 (comment)

[jordanhendricks]

I've begun this work, but am not going to have a PR ready before I go on PTO for ~1.5 weeks.

The design we've reached consensus on is that the swap device will be an encrypted zvol (with an ephemeral key) on the zpool of the M.2 the sled booted from.

We will create the device as such (though the actual zfs commands will be run by sled-agent):

# starting with 256GB until testing proves otherwise, but will make this configurable via sled-agent manifest
size=<from sled-agent manifest>
blocksize=`getconf PAGESIZE`

/sbin/zfs create \
        # creates a sparse volume with no reservation
        -s \

        # volume of a given size; exported as a block device in /dev/zvol/{dsk,rdsk}/<path>
        -V $size \

        # equivalent to volblocksize=blocksize
        -b $blocksize \

        # -o property=value
        #
        # don't use log device
        -o logbias=throughput \

        # only cache metadata in ARC
        -o primarycache=metadata \

        # don't use secondary cache
        -o secondarycache=none \

        # encryption format
        -o encryption=aes-256-gcm \

        # raw key (32 bytes)
        -o keyformat=raw \

        # path to key
        -o keylocation=file://path/to/key \

        # name of device
        oxi_<M.2 zpool uuid>/swap

The device will be added via swapctl(2).

Because the size the device should be is still an open question, we may need to resize it if it already exists. Thus, on startup, sled-agent will do the following:

• check if there is a swap device, and if so, whether it is a zvol of the right size
• if the zvol exists, but is the wrong size, resize it; then give the existing zvol a new key, and add it as a swap device if it's not already
• if there is no swap device, then create the zvol and add it via swapctl