allow sled agent to configure a swap device #3571

jordanhendricks · 2023-07-12T08:17:32Z

Summary

When configured via sled-agent's config.toml via the parameter swap_device_size_gb, sled agent will create an encrypted zvol of the specified size, using an ephemeral key for encryption, on the zpool of the M.2 that the sled booted from. It will then use that zvol as a swap device. (See the comment for ensure_swap_device for a discussion of how this creation is made safe in the face of crashes/restarts of the agent and reboots of the sled).

The default value for the gimlet config is 256 GiB. I got that value from some discussion somewhere, but I need to find that reasoning and record it.

Fixes #2858

Testing Notes

I tested this on sn05 using these instructions to get going.

Note about the swap -l and log line output below: swapctl(2) returns the start of the "swapfile", and its length, in 512-byte blocks. So in the examples below, 536870904 blocks * 512 ~= 256 GiB. The start field starts 8 blocks in because the first page of the file is reserved.

fresh install: no swap devices, zvol doesn't exist

environment before:

EVT22200005 # zfs list | grep swap 
EVT22200005 # swap -l
No swap devices configured

Installing omicron from a fresh slate we see these log lines:

07:57:49.063Z INFO SledAgent: Requested swap device of size 256 GiB
    sled_id = 1b6e8794-0313-44c8-bc25-f18860205d50
07:57:49.113Z INFO SledAgent: attempting to create encrypted zvol: name="oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap", size_gb=256
07:57:49.193Z INFO SledAgent: successfully created encrypted zvol: name="oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap", size_gb=256
07:57:49.193Z INFO SledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap"

The device and associated zvol are there:

EVT22200005 # swap -l
swapfile             dev    swaplo   blocks     free
/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap 170,5         8 536870904 536870904
EVT22200005 # zfs list -r -o name,type,used,avail oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5
NAME                                              TYPE         USED  AVAIL
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5          filesystem  1.77M   725G
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/cluster  filesystem    96K   725G
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/config   filesystem   156K   725G
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/crash    filesystem    96K   725G
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/debug    filesystem    96K   100G
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/install  filesystem    96K   725G
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap     volume        88K   725G

sled-agent restart: swap device exists

Building off of the previous environment, now we restart sled-agent via svadm restart sled-agent. We see that sled-agent detected the swap device and moved on:

08:00:19.010Z INFO SledAgent: Requested swap device of size 256 GiB
    sled_id = 1b6e8794-0313-44c8-bc25-f18860205d50
08:00:19.010Z INFO SledAgent: Swap device already exists: [SwapDevice { path: "/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap", start: 8, length: 536870904, total_pages: 67108863, free_pages: 67108863, flags: 0 }]

sled-agent restart: swap device doesn't exist, but zvol does

Next, remove the swap device and restart sled-agent again to ensure that it makes a new swap device.

Before:

EVT22200005 # swap -l
swapfile             dev    swaplo   blocks     free
/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap 170,5         8 536870904 536870904

Capture the guid and creation time of the zvol so we can make sure it gets recreated:

EVT22200005 # zfs list -o name,type,guid,creation oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap                                                           
NAME                                           TYPE     GUID  CREATION
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap  volume  13421629411158969258  Wed Jul 12  8:01 2023

Delete the swap device and restart:

EVT22200005 # swap -d /dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap
EVT22200005 # swap -l
No swap devices configured
EVT22200005 # svcadm restart sled-agent

We see the sled agent deleted the old zvol and created a new one:

08:07:35.217Z INFO SledAgent: Requested swap device of size 256 GiB
    sled_id = 1b6e8794-0313-44c8-bc25-f18860205d50
08:07:35.269Z INFO SledAgent: swap zvol "oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap" aleady exists; destroying
08:07:35.346Z INFO SledAgent: swap zvol "oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap" destroyed
08:07:35.346Z INFO SledAgent: attempting to create encrypted zvol: name="oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap", size_gb=256
08:07:35.422Z INFO SledAgent: successfully created encrypted zvol: name="oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap", size_gb=256
08:07:35.422Z INFO SledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap"

That's confirmed by the zvol guid and creation time:

EVT22200005 # zfs list -o name,type,guid,creation oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap 
NAME                                           TYPE     GUID  CREATION
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap  volume  1488254613385143009  Wed Jul 12  8:07 2023

And we see a swap device again:

EVT22200005 # swap -l
swapfile             dev    swaplo   blocks     free
/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap 170,7         8 536870904 536870904

sled-agent restart: neither swap device nor zvol exists

This is the same behavior s the fresh install case, but here I delete the swap device and zvol manually, then restart sled agent:

EVT22200005 # swap -l                                                        
swapfile             dev    swaplo   blocks     free                                                                                                      
/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap 170,7         8 536870904 536870904
EVT22200005 #                                                                                                                                             
EVT22200005 # swap -l                                                                                                                                     
swapfile             dev    swaplo   blocks     free                 
/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap 170,7         8 536870904 536870904
EVT22200005 # zfs list -o name,type,guid,creation oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap
NAME                                           TYPE     GUID  CREATION
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap  volume  1488254613385143009  Wed Jul 12  8:07 2023
EVT22200005 # swap -d /dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap
EVT22200005 # zfs destroy oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap
EVT22200005 # swap -l                                                        
No swap devices configured         
     EVT22200005 # zfs list oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap
cannot open 'oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap': dataset does not exist
EVT22200005 # svcadm restart sled-agent

After the restart, everything looks good:

08:11:53.051Z INFO SledAgent: Requested swap device of size 256 GiB
    sled_id = 1b6e8794-0313-44c8-bc25-f18860205d50
08:11:53.103Z INFO SledAgent: attempting to create encrypted zvol: name="oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap", size_gb=256
08:11:53.181Z INFO SledAgent: successfully created encrypted zvol: name="oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap", size_gb=256
08:11:53.181Z INFO SledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap"

EVT22200005 # swap -l
swapfile             dev    swaplo   blocks     free
/dev/zvol/dsk/oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap 170,8         8 536870904 536870904
EVT22200005 # zfs list -o name,type,guid,creation oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap
NAME                                           TYPE     GUID  CREATION
oxi_d95aadb8-e17c-4feb-9c5d-d5a030ae35a5/swap  volume  9578766136324772467  Wed Jul 12  8:11 2023

sled agent not configured for swap

This is useful for circumstances in which we don't want sled-agent to configure a swap device, such as a non-gimlet setup (helios configures a device by default). To test this, remove swap_device_size_gb from the config:

08:14:29.649Z INFO SledAgent: Not setting up swap device: not configured
    sled_id = 1b6e8794-0313-44c8-bc25-f18860205d50

smklein

This looks great -- the code is extremely readable, and I'm thrilled to have this integrated!

smf/sled-agent/gimlet/config.toml

sled-agent/src/swap_device.rs

smklein · 2023-07-12T08:34:05Z

sled-agent/src/swap_device.rs

+/// The swap device is an encrypted zvol that lives on the M.2 disk that the
+/// system booted from.  Because it booted from the disk, we know for certain
+/// the system can access it. We encrypt the zvol because arbitrary system
+/// memory could exist in swap, including sensitive data. The zvol is encrypted


(No action necessary) I totally think this is the right call, but in a parallel vein -- aren't we going to try to set up a dump device too? That also seems like an area where we might "put the OS's memory onto disk" but I don't think we had plans to encrypt it. Maybe we should?

Hm, I forget what exactly we decided here. I see that @faithanalog is assigned to #2450. Do you have any context there?

Otherwise, I need to see if I can find the recording of the meeting we had for the design of the swap device and see if we discussed it there.

Yeah, definitely no action needed from this PR, but mostly just a musing, since dump and swap are comparable for "what memory is written to disk".

sled-agent/src/swap_device.rs

citrus-it

The approach, logic and the way the device is being configured all look good to me. Thanks for doing this.

citrus-it · 2023-07-12T08:52:03Z

sled-agent/src/sled_agent.rs

+                )?;
+            }
+            Some(sz) if sz == 0 => {
+                panic!("Invalid requested swap device size of 0 GiB");


Your notes say that it's ok to set the size to 0

To test this, remove swap_device_size_gb (or set it to 0):

one or t'other should be changed.

Oops! Good catch. I don't really want to change my approach in the code so I updated my notes.

…nitial PR

gjcolombo

LGTM, just a couple of nits. Thanks for putting this together!

gjcolombo · 2023-07-12T15:48:36Z

sled-agent/src/swap_device.rs

+pub(crate) fn ensure_swap_device(
+    log: &slog::Logger,
+    boot_zpool_name: &illumos_utils::zpool::ZpoolName,
+    size_gb: u32,


nit: maybe use NonZeroU32 here? I'm not sure how much uglier that'll make the caller, though.

I think I'd rather punt on this, if that's okay.

gjcolombo · 2023-07-12T15:52:02Z

sled-agent/src/swap_device.rs

+        info!(log, "swap zvol \"{}\" destroyed", swap_zvol);
+    }
+
+    // The process of paging out using block I/O, so use the "dsk" version of


nit: s/using/uses?

done in b739a72

gjcolombo · 2023-07-12T16:28:08Z

sled-agent/src/swap_device.rs

+    Ok(())
+}
+
+// Check whether the given zvol exists.


I'd make this comment and the other private-function comments into doc comments even though the functions aren't pub (my dev environment, and I assume other similar IDEs, will only show these comments on hover if they're doc comments).

done in b739a72

allow sled agent to configure a swap device

33a6b2b

jordanhendricks requested review from citrus-it, gjcolombo and smklein July 12, 2023 08:17

jordanhendricks added this to the FCS milestone Jul 12, 2023

make clippy happy (I think); remove todos from testing

a3592a8

smklein approved these changes Jul 12, 2023

View reviewed changes

citrus-it reviewed Jul 12, 2023

View reviewed changes

jordanhendricks added 5 commits July 12, 2023 15:34

Review feedback

2d27a13

Add license to newly added file

43c4ba7

Fix build errors

299d4a2

Remove duplicated comment

03d529a

Add in hunk to correctly use zvol/blk paths that did not make it in i…

6fb3c86

…nitial PR

gjcolombo approved these changes Jul 12, 2023

View reviewed changes

jordanhendricks added 3 commits July 12, 2023 16:42

More review feedback

b739a72

clippy

4880fb2

Fix linux build

b9c78db

jordanhendricks merged commit 127db25 into oxidecomputer:main Jul 12, 2023

jordanhendricks deleted the swap-device branch July 12, 2023 23:36

jordanhendricks mentioned this pull request Jul 16, 2023

omicron-package uninstall can fail deleting swapfile #3651

Closed

iliana mentioned this pull request Oct 11, 2023

tools/destroy_virtual_hardware.sh is missing some new steps #4245

Open

allow sled agent to configure a swap device #3571

allow sled agent to configure a swap device #3571

Uh oh!

Conversation

jordanhendricks commented Jul 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing Notes

Uh oh!

smklein left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

citrus-it left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gjcolombo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jordanhendricks commented Jul 12, 2023 •

edited

Loading