crypt/debug dataset not mounted

[jmpesp]

On dogfood, we found that many pool's crypt/debug datasets were not mounted:

# zfs list -r -o name,mountpoint,mounted oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt
NAME                                                                                                          MOUNTPOINT                                                                                                          MOUNTED
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt                                                                /pool/ext/4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt                                                                    yes
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/debug                                                          /pool/ext/4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/debug                                                               no
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone                                                           /pool/ext/4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone                                                               yes
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone/oxz_crucible_ce8563f3-4a93-45ff-b727-cbfbee6aa413         /pool/ext/4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone/oxz_crucible_ce8563f3-4a93-45ff-b727-cbfbee6aa413             yes
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone/oxz_propolis-server_7bbb541b-6732-4d42-b03c-c7a5fd7c0c19  /pool/ext/4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone/oxz_propolis-server_7bbb541b-6732-4d42-b03c-c7a5fd7c0c19      yes

At the minimum this resulted in rotated logs not being compressed, and the storage for those logs ending up in the parent crypt dataset:

BRM44220005 # zfs list -r -o name,avail,used oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt
NAME                                                                                                          AVAIL   USED
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt                                                                1.84T   441G
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/debug                                                           100G    99K
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone                                                           1.84T   610M
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone/oxz_crucible_ce8563f3-4a93-45ff-b727-cbfbee6aa413         1.84T   299M
oxp_4358f47f-f21e-4cc8-829e-0c7fc2400a59/crypt/zone/oxz_propolis-server_7bbb541b-6732-4d42-b03c-c7a5fd7c0c19  1.84T   312M

[leftwo]

Many = All in dogfood:

root@oxz_switch0:~# pilot host exec -c 'hostname && zfs list -r -o name,mountpoint,mounted | grep crypt/debug | grep yes' 0-31
 7  BRM27230045        failure: exit code 1: BRM27230045
 8  BRM44220011        failure: exit code 1: BRM44220011
 9  BRM44220005        failure: exit code 1: BRM44220005
10  BRM42220009        failure: exit code 1: BRM42220009
11  BRM42220006        failure: exit code 1: BRM42220006
12  BRM42220057        failure: exit code 1: BRM42220057
13  BRM42220018        failure: exit code 1: BRM42220018
14  BRM42220051        failure: exit code 1: BRM42220051
16  BRM42220014        failure: exit code 1: BRM42220014
17  BRM42220017        failure: exit code 1: BRM42220017
21  BRM42220031        failure: exit code 1: BRM42220031
23  BRM42220016        failure: exit code 1: BRM42220016
25  BRM44220010        failure: exit code 1: BRM44220010
29  BRM42220063        failure: exit code 1: BRM42220063

No yes in that output, Not a single .../crypt/debug is mounted on dogfood.

[bnaecker]

I'm not paged in on this most recent occurrence, but seems like this is at least related to #4203.

[askfongjojo]

The PR #3735 (implemented slightly after the first customer ship) includes a change from using a directory to a dataset. I don't know if that might explain this issue. If so, we probably will see this same problem at the customer deployed before the change.

[leftwo]

On a fresh install on a racklette, I have crypt/debug mounted:

root@oxz_switch0:~# pilot host exec -c 'hostname && zfs list -r -o name,mountpoint,mounted | grep crypt/debug | grep yes' 0-31                                                                                                                                            
14  BRM42220026        ok: BRM42220026                                                                                                                                                                                                                                    
oxp_0aaf7d20-ca6c-44ae-87da-463206f1bd2c/crypt/debug                                                          /pool/ext/0aaf7d20-ca6c-44ae-87da-463206f1bd2c/crypt/debug                                                              yes                                 
oxp_15288972-8d11-45b4-889f-ab3cfdfa0ca2/crypt/debug                                                          /pool/ext/15288972-8d11-45b4-889f-ab3cfdfa0ca2/crypt/debug                                                              yes                                 
oxp_725732c7-6181-4e25-a54c-ea30efddca06/crypt/debug                                                          /pool/ext/725732c7-6181-4e25-a54c-ea30efddca06/crypt/debug                                                              yes                                 
oxp_738d39f9-7951-4cba-9e1d-fd9738f5c640/crypt/debug                                                          /pool/ext/738d39f9-7951-4cba-9e1d-fd9738f5c640/crypt/debug                                                              yes                                 
oxp_b1b060b3-cfd8-47f3-be86-16b952d7b5f9/crypt/debug                                                          /pool/ext/b1b060b3-cfd8-47f3-be86-16b952d7b5f9/crypt/debug                                                              yes                                 
oxp_c9048c5d-1476-4d9b-8793-27a92c2285f7/crypt/debug                                                          /pool/ext/c9048c5d-1476-4d9b-8793-27a92c2285f7/crypt/debug                                                              yes                                 
oxp_cd7849dc-96d6-4077-85ae-1f9caffcf1e4/crypt/debug                                                          /pool/ext/cd7849dc-96d6-4077-85ae-1f9caffcf1e4/crypt/debug                                                              yes                                 
oxp_e10d305c-afd8-47b3-9f55-7fc38bbb4171/crypt/debug                                                          /pool/ext/e10d305c-afd8-47b3-9f55-7fc38bbb4171/crypt/debug                                                              yes                                 
oxp_e22a4d0d-4575-4a7a-904d-81885a1f433b/crypt/debug                                                          /pool/ext/e22a4d0d-4575-4a7a-904d-81885a1f433b/crypt/debug                                                              yes                                 
15  BRM27230037        ok: BRM27230037                                                                                                                                                                                                                                    
oxp_0d27b98f-c0ca-4bea-89a6-350ca2ec3eb3/crypt/debug                                                       /pool/ext/0d27b98f-c0ca-4bea-89a6-350ca2ec3eb3/crypt/debug                                                           yes                                       
oxp_264b01e2-b0e6-4485-9834-36ed9626096d/crypt/debug                                                       /pool/ext/264b01e2-b0e6-4485-9834-36ed9626096d/crypt/debug                                                           yes                                       
oxp_52125f61-6516-4ddc-94f8-91d93a4272c8/crypt/debug                                                       /pool/ext/52125f61-6516-4ddc-94f8-91d93a4272c8/crypt/debug                                                           yes                                       
oxp_54347074-63bb-4c82-94cf-b8488dcda4e3/crypt/debug                                                       /pool/ext/54347074-63bb-4c82-94cf-b8488dcda4e3/crypt/debug                                                           yes                                       
oxp_5efb2817-506a-43df-aaca-b0fe325816ed/crypt/debug                                                       /pool/ext/5efb2817-506a-43df-aaca-b0fe325816ed/crypt/debug                                                           yes                                       
oxp_87b7322b-6d2d-46e6-ab2d-fbfc068a23ad/crypt/debug                                                       /pool/ext/87b7322b-6d2d-46e6-ab2d-fbfc068a23ad/crypt/debug                                                           yes                                       
oxp_94cd77d1-7336-45c2-b508-f2096288ddb2/crypt/debug                                                       /pool/ext/94cd77d1-7336-45c2-b508-f2096288ddb2/crypt/debug                                                           yes                                       
oxp_c4acfbb1-8a55-4ebf-845f-954d9ebd4c00/crypt/debug                                                       /pool/ext/c4acfbb1-8a55-4ebf-845f-954d9ebd4c00/crypt/debug                                                           yes                                       
oxp_df3a5581-7721-4704-8421-1d283e428e0c/crypt/debug                                                       /pool/ext/df3a5581-7721-4704-8421-1d283e428e0c/crypt/debug                                                           yes                                       
oxp_e8129e95-1ae0-4ef4-bfc3-0fad4af00761/crypt/debug                                                       /pool/ext/e8129e95-1ae0-4ef4-bfc3-0fad4af00761/crypt/debug                                                           yes                                       
16  BRM23230018        ok: BRM23230018                                                                                                                                                                                                                                    
oxp_0938306f-dde7-4bd9-9267-fae5f056be5b/crypt/debug                                                          /pool/ext/0938306f-dde7-4bd9-9267-fae5f056be5b/crypt/debug                                                              yes                                 
oxp_23e587a0-4f69-49d2-bb7a-5f463a2a7833/crypt/debug                                                          /pool/ext/23e587a0-4f69-49d2-bb7a-5f463a2a7833/crypt/debug                                                              yes                                 
oxp_36f6fe2b-6f56-4de2-b426-493cd8d45852/crypt/debug                                                          /pool/ext/36f6fe2b-6f56-4de2-b426-493cd8d45852/crypt/debug                                                              yes                                 
oxp_554f1065-bf7b-48ec-9226-8952d2ec0630/crypt/debug                                                          /pool/ext/554f1065-bf7b-48ec-9226-8952d2ec0630/crypt/debug                                                              yes                                 
oxp_62133092-6fa0-41ad-a0e1-3afd43b37351/crypt/debug                                                          /pool/ext/62133092-6fa0-41ad-a0e1-3afd43b37351/crypt/debug                                                              yes                                 
oxp_c1c447b3-a767-4a8d-8fd6-9a91e3269f0c/crypt/debug                                                          /pool/ext/c1c447b3-a767-4a8d-8fd6-9a91e3269f0c/crypt/debug                                                              yes                                 
oxp_dda9b76c-6fc6-4ba5-89c4-cde76d46802d/crypt/debug                                                          /pool/ext/dda9b76c-6fc6-4ba5-89c4-cde76d46802d/crypt/debug                                                              yes                                 
oxp_df63ecb0-a61c-48ea-8865-b7b0a3f8a94f/crypt/debug                                                          /pool/ext/df63ecb0-a61c-48ea-8865-b7b0a3f8a94f/crypt/debug                                                              yes                                 
oxp_fda6309e-fed9-4a37-af05-5f03b5247639/crypt/debug                                                          /pool/ext/fda6309e-fed9-4a37-af05-5f03b5247639/crypt/debug                                                              yes                                 
17  BRM23230010        ok: BRM23230010                                                                                                                                                                                                                                    
oxp_169f6d4e-002d-444a-8363-18538177830c/crypt/debug                                                          /pool/ext/169f6d4e-002d-444a-8363-18538177830c/crypt/debug                                                              yes                                 
oxp_1fe56533-72a0-43b9-80ea-6d3ecfc3e23e/crypt/debug                                                          /pool/ext/1fe56533-72a0-43b9-80ea-6d3ecfc3e23e/crypt/debug                                                              yes                                 
oxp_25209c9b-5c44-4ba3-b9d3-9a93c3975fe7/crypt/debug                                                          /pool/ext/25209c9b-5c44-4ba3-b9d3-9a93c3975fe7/crypt/debug                                                              yes                                 
oxp_2ca70b18-f09f-406f-895c-1aef7d79de3b/crypt/debug                                                          /pool/ext/2ca70b18-f09f-406f-895c-1aef7d79de3b/crypt/debug                                                              yes                                 
oxp_31483530-0ca9-4988-8a70-7736c40cf9b7/crypt/debug                                                          /pool/ext/31483530-0ca9-4988-8a70-7736c40cf9b7/crypt/debug                                                              yes                                 
oxp_52164519-a524-403c-acc6-c957275344a8/crypt/debug                                                          /pool/ext/52164519-a524-403c-acc6-c957275344a8/crypt/debug                                                              yes                                 
oxp_d296f76d-139d-4e56-af50-1f4cd430f899/crypt/debug                                                          /pool/ext/d296f76d-139d-4e56-af50-1f4cd430f899/crypt/debug                                                              yes                                 
oxp_eb84dea8-442a-4c96-bd43-b041ac4877cc/crypt/debug                                                          /pool/ext/eb84dea8-442a-4c96-bd43-b041ac4877cc/crypt/debug                                                              yes                                 
oxp_fba51b28-bfb3-4335-ab9a-86207c039bf7/crypt/debug                                                          /pool/ext/fba51b28-bfb3-4335-ab9a-86207c039bf7/crypt/debug                                                              yes                                 
oxp_ff0dfd63-3f1b-4544-be64-ece53e65550d/crypt/debug                                                          /pool/ext/ff0dfd63-3f1b-4544-be64-ece53e65550d/crypt/debug                                                              yes                 

I parked the rack, then rebooted all the sleds, and when things came up, no more crypt/debug:

root@oxz_switch0:~# pilot host exec -c 'hostname && zfs list -r -o name,mountpoint,mounted | grep crypt/debug | grep yes' 0-31
14  BRM42220026        failure: exit code 1: BRM42220026
15  BRM27230037        failure: exit code 1: BRM27230037
16  BRM23230018        failure: exit code 1: BRM23230018
17  BRM23230010        failure: exit code 1: BRM23230010
Error: some operations failed

[leftwo]

I did a clean slate, then new RSS on a racklette. After creating a project and starting an instance, I rebooted just one sled.
When that sled came back up, it no longer had .../crypt/debug datasets mounted.

[smklein]

There are two spots where we "should" be ensuring the debug dataset gets mounted.

1. ensure_zpool_has_datasets. This is called on all disks as they are initialized by the sled agent - we should be ensuring that "expected" datasets exist on U.2s and M.2s. This includes the debug dataset. This is one of the "bootstrapping" phases of disk initialization, before Nexus has even made any requests.

2. When we call omicron_config_put, and eventually write the config for explicitly-controlled datasets. This should write configuration for "datasets we want", which the sled agent should then mount on reboot. See: https://github.com/oxidecomputer/omicron/blob/main/sled-agent/src/sled_agent.rs#L858 . Since Nexus is aware of debug datasets, this configuration should include debug datasets, and sled agent should have enough context to explicitly mount them.

Why are we not mounting during ensure_zpool_has_datasets?

From:

omicron/sled-storage/src/dataset.rs

Line 191 in 5fd1c35

pub(crate) async fn ensure_zpool_has_datasets(

• datasets, from U2_EXPECTED_DATASETS, does include crypt/debug
• Our first call to ensure_dataset is on the root - crypt - , and does import it:

  omicron/sled-storage/src/dataset.rs

  Line 265 in 5fd1c35

  let result = Zfs::ensure_dataset(zfs::DatasetEnsureArgs {

• We then iterate over individual datatasets within the root (e.g., zones, debug), and delete them if needed:

  omicron/sled-storage/src/dataset.rs

  Line 282 in 5fd1c35

  for dataset in datasets.into_iter() {

• We do call ensure_dataset on these individual datasets:

  omicron/sled-storage/src/dataset.rs

  Lines 321 to 335 in 5fd1c35

  	let encryption_details = None;
  	let size_details = Some(SizeDetails {
  	quota: dataset.quota,
  	reservation: None,
  	compression: dataset.compression,
  	});
  	Zfs::ensure_dataset(zfs::DatasetEnsureArgs {
  	name,
  	mountpoint: Mountpoint::Path(mountpoint),
  	zoned,
  	encryption_details,
  	size_details,
  	id: None,
  	additional_options: None,
  	})?;

• HOWEVER, we pass encryption_details = None:

  omicron/sled-storage/src/dataset.rs

  Line 321 in 5fd1c35

  let encryption_details = None;

• For the zones dataset - which we just deleted - we'll recreate it, and mount it. This means: exists = false, mounted = false, encryption_details = None
• For the debug dataset, which we did not delete, we should see: exists = true, mounted = false, encryption_details = None

This means the debug dataset will hit this branch:

omicron/illumos-utils/src/zfs.rs

Lines 595 to 597 in 5fd1c35

	if encryption_details.is_none() {
	// If the dataset exists, we're done. Unencrypted datasets are
	// automatically mounted.

This returns early, and makes no attempt to mount the debug dataset.

[smklein]

Crossposting from chat for posterity: it's possible to use the "immutable" attribute on directories to ensure they can only be used as mountpoints. This would help us avoid the problem of "someone wrote to debug when it was a directory, not a mountpoint":

$ mkdir foobar
$ pfexec chmod S+ci foobar
$ touch foobar/hi
touch: cannot create foobar/hi: Not owner