blueprint_executor returned Zpool Not Found error when processing an expunged disk #7500
Description
I ran into this issue after upgrading the dublin racklet to a relatively recent omicron commit and attempted to expunge a disk that has a crucible and a pantry zone. The rack hasn't gone through any blueprint update previously.
root@oxz_switch0:~# pilot host ls
CUBBY IP SERIAL IMAGE
14 fe80::aa40:25ff:fe04:351 BRM42220026 ci bbcadc5/edb91c0 2025-02-06 16:07
15 fe80::aa40:25ff:fe04:614 BRM27230037 ci bbcadc5/edb91c0 2025-02-06 16:07
16 fe80::aa40:25ff:fe04:6d3 BRM23230018 ci bbcadc5/edb91c0 2025-02-06 16:07
17 fe80::aa40:25ff:fe04:851 BRM23230010 ci bbcadc5/edb91c0 2025-02-06 16:07
root@oxz_switch0:~# omdb nexus blueprints list
T ENA ID PARENT TIME_CREATED
* no f0c4f78e-bf43-410e-ab7b-9448de4044ed <none> 2025-02-06T02:41:12.512Z
root@oxz_switch0:~# omdb -w nexus sleds expunge-disk 720e2adf-a837-49af-9015-68af73122654
WARNING: physical disk 720e2adf-a837-49af-9015-68af73122654 is PRESENT in the most recent inventory collection (spotted at 2025-02-06 19:40:08.439074 UTC). Although expunging a running disk is supported, it is safer to expunge a disk from a system where it has been removed. Are you sure you want to proceed anyway?
y/N〉y
WARNING: This operation will PERMANENTLY and IRRECOVABLY mark physical disk 720e2adf-a837-49af-9015-68af73122654 (A079DDE7) expunged. To proceed, type the physical disk's serial number.
disk serial number〉A079DDE7
expunged disk 720e2adf-a837-49af-9015-68af73122654
root@oxz_switch0:~# omdb -w nexus blueprints regenerate
generated new blueprint 4645bedc-1160-4100-8ea7-8082e3b18954
root@oxz_switch0:~# omdb -w nexus blueprints diff current 4645bedc-1160-4100-8ea7-8082e3b18954
from: blueprint f0c4f78e-bf43-410e-ab7b-9448de4044ed
to: blueprint 4645bedc-1160-4100-8ea7-8082e3b18954
UNCHANGED SLEDS:
sled c2cc9458-eb14-42f3-847d-6842815999a1 (active):
physical disks at generation 2:
-----------------------------------
vendor model serial
-----------------------------------
1b96 WUS4C6432DSP3X3 A079DDCD
1b96 WUS4C6432DSP3X3 A079DF2B
1b96 WUS4C6432DSP3X3 A079DF42
1b96 WUS4C6432DSP3X3 A079DF5A
1b96 WUS4C6432DSP3X3 A079DF5C
1b96 WUS4C6432DSP3X3 A079DFFB
1b96 WUS4C6432DSP3X3 A079E3AC
1b96 WUS4C6432DSP3X3 A079E3D6
1b96 WUS4C6432DSP3X3 A079E435
1b96 WUS4C6432DSP3X3 A079E4A7
datasets at generation 1:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name dataset uuid quota reservation compression
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_100a2253-5000-4841-9788-2479abb6e034/crypt/cockroachdb 9ce15ae2-fa4f-44d7-aa37-beb74244d475 none none off
oxp_f4e65604-4406-47f6-a250-a23c8bfd03f8/crypt/cockroachdb ab63b6bd-0689-4ff5-8136-695291d38401 none none off
oxp_0130891e-1ff3-4949-9f7b-5ecfb15dfce4/crucible b25ba62e-5ebc-404c-9621-28903abaf400 none none off
oxp_0d566b88-9861-4a74-bbe4-fc1ac425bba0/crucible dfd006dd-0aaf-4205-acab-069876399556 none none off
oxp_100a2253-5000-4841-9788-2479abb6e034/crucible 17fb4697-5d4a-428f-9378-34976022ecfc none none off
oxp_434b76e9-ff3a-4621-9824-54f49ccbd49a/crucible 8545b9e5-1afe-4b95-aa7d-b587bdc669c5 none none off
oxp_5dae56a9-dffc-4362-bed3-bd83fa3d11d4/crucible 4b982aaa-789d-47be-a5a5-52f6b8d8d8bb none none off
oxp_9e6842e5-f930-44bd-929b-5a009715f0f7/crucible c4237a79-7439-4a8e-8bf1-f6c8c928c828 none none off
oxp_b61ccadb-aa41-4e07-8650-ae4989e40f46/crucible 80327fff-5c85-48e6-abb6-51a47b9520c4 none none off
oxp_d242a551-c4bf-472c-a155-b28b55f1a089/crucible 34056d9c-1dd1-41c4-8a99-cfc4d968a421 none none off
oxp_f4e65604-4406-47f6-a250-a23c8bfd03f8/crucible e3b93223-fd93-4249-8976-c872bd4a950e none none off
oxp_fbb0a63a-abef-4d79-b2c5-bbdfaf501319/crucible 8bc8d8cf-07e1-45cd-8374-d9077c896429 none none off
oxp_0130891e-1ff3-4949-9f7b-5ecfb15dfce4/crypt/zone 2b1e4c6e-047b-4b23-b7b5-4aa159db313c none none off
oxp_0d566b88-9861-4a74-bbe4-fc1ac425bba0/crypt/zone fc42ae29-21ab-440d-ae4d-c59bb56d2f9d none none off
oxp_100a2253-5000-4841-9788-2479abb6e034/crypt/zone 83cf2028-8cc9-4ad4-a8b1-a6b55662ae7c none none off
oxp_434b76e9-ff3a-4621-9824-54f49ccbd49a/crypt/zone 1c0d140e-949d-425f-a6df-3f86720f24b7 none none off
oxp_5dae56a9-dffc-4362-bed3-bd83fa3d11d4/crypt/zone e7f7756d-e52e-48b1-bce1-de04d4053085 none none off
oxp_9e6842e5-f930-44bd-929b-5a009715f0f7/crypt/zone b4a4d586-c28d-4cde-bc36-019025088a1a none none off
oxp_b61ccadb-aa41-4e07-8650-ae4989e40f46/crypt/zone 4e6f0736-2e30-437c-ba22-0fa3f9a7e2c2 none none off
oxp_d242a551-c4bf-472c-a155-b28b55f1a089/crypt/zone 6f7ce7bb-0e75-4f6a-87aa-8185645382b7 none none off
oxp_f4e65604-4406-47f6-a250-a23c8bfd03f8/crypt/zone 22dabf5a-bb6d-4c5d-97f2-f388363b5afd none none off
oxp_fbb0a63a-abef-4d79-b2c5-bbdfaf501319/crypt/zone fa9e34c5-6fce-47c2-a74b-7b87cf90e649 none none off
oxp_f4e65604-4406-47f6-a250-a23c8bfd03f8/crypt/zone/oxz_cockroachdb_bed54974-27d1-4425-bae1-8eddde89dea7 e4579ae4-532e-47b3-9fdf-6416873e1ef4 none none off
oxp_100a2253-5000-4841-9788-2479abb6e034/crypt/zone/oxz_cockroachdb_f98990d5-2f50-4c7a-b816-bbdb807c4712 c9590417-c4b8-418e-938d-857c2b5042f4 none none off
oxp_f4e65604-4406-47f6-a250-a23c8bfd03f8/crypt/zone/oxz_crucible_0ffe7a0a-7167-4ecd-ac54-7f8b057a92f0 a66e9628-87ff-4591-8860-aaf6962b116a none none off
oxp_fbb0a63a-abef-4d79-b2c5-bbdfaf501319/crypt/zone/oxz_crucible_2ddda7b9-e995-4a83-a944-5c9d8c25a76d 8b457cbd-0817-4032-8298-fd221e4723e3 none none off
oxp_9e6842e5-f930-44bd-929b-5a009715f0f7/crypt/zone/oxz_crucible_41fc8d59-c462-40c8-bec2-d3d4ae1eb928 68921128-952a-43d8-847e-d4f9084a900c none none off
oxp_434b76e9-ff3a-4621-9824-54f49ccbd49a/crypt/zone/oxz_crucible_507e77a0-3027-4d37-81e9-80e5db1f0ae8 1836997a-03f0-47fa-9754-2eb923d6422b none none off
oxp_5dae56a9-dffc-4362-bed3-bd83fa3d11d4/crypt/zone/oxz_crucible_7b3cf3b6-9fbf-4309-9920-77715ca37b9a 51711b9c-da34-4851-8eae-dc65c3dd8d30 none none off
oxp_100a2253-5000-4841-9788-2479abb6e034/crypt/zone/oxz_crucible_b42afe67-c0f6-4a09-b323-08755282e497 027fe627-19eb-4887-9a3b-6933dac882a3 none none off
oxp_0130891e-1ff3-4949-9f7b-5ecfb15dfce4/crypt/zone/oxz_crucible_c5e350f6-15f8-4754-a478-20992d0a0b46 17efbff9-cab4-4cd1-a97c-65fa6fce4b18 none none off
oxp_b61ccadb-aa41-4e07-8650-ae4989e40f46/crypt/zone/oxz_crucible_d5784af6-bcd4-4ace-b399-baad8557136d 85242943-1ac8-45c5-9c44-1f16a2824919 none none off
oxp_d242a551-c4bf-472c-a155-b28b55f1a089/crypt/zone/oxz_crucible_ef1a78c5-1afc-4a02-a7b3-5eaab530e6f3 1fc63806-7564-414e-ad4e-8ae0f2519bbf none none off
oxp_0d566b88-9861-4a74-bbe4-fc1ac425bba0/crypt/zone/oxz_crucible_f33d73b0-0838-408e-85e0-847a73a4b9a3 39aa1fed-2b82-41f8-b285-5b621ac1f40e none none off
oxp_0d566b88-9861-4a74-bbe4-fc1ac425bba0/crypt/zone/oxz_crucible_pantry_80d2093d-6d34-4a18-8ee1-964c60e8b8d3 015afd07-3462-47cb-a59a-90122c951aee none none off
oxp_9e6842e5-f930-44bd-929b-5a009715f0f7/crypt/zone/oxz_nexus_f19832c3-8f7c-4cd6-bcbd-dc0ef888e11a a16661dc-3b5d-4af0-b774-062dc01de09c none none off
oxp_d242a551-c4bf-472c-a155-b28b55f1a089/crypt/zone/oxz_ntp_6dfeb901-8409-43e3-84ff-86aebe7e0d8a cb397eb6-45a8-4075-8a55-a52f0ec32f36 none none off
oxp_0130891e-1ff3-4949-9f7b-5ecfb15dfce4/crypt/debug 4deb81cb-6de3-4b1c-bfdf-a8defee262fb 100 GiB none gzip-9
oxp_0d566b88-9861-4a74-bbe4-fc1ac425bba0/crypt/debug a070cf81-1ba5-4b84-859d-14468c619e8c 100 GiB none gzip-9
oxp_100a2253-5000-4841-9788-2479abb6e034/crypt/debug 8ef88269-ea81-4b9f-963a-00c34c71f24a 100 GiB none gzip-9
oxp_434b76e9-ff3a-4621-9824-54f49ccbd49a/crypt/debug bd26a319-b922-44df-a514-1e7f10e1a159 100 GiB none gzip-9
oxp_5dae56a9-dffc-4362-bed3-bd83fa3d11d4/crypt/debug 14f0f104-eeed-4950-8ed9-2fd8427150ed 100 GiB none gzip-9
oxp_9e6842e5-f930-44bd-929b-5a009715f0f7/crypt/debug a3bda47b-045e-4df5-8bf0-3942502e5c10 100 GiB none gzip-9
oxp_b61ccadb-aa41-4e07-8650-ae4989e40f46/crypt/debug f01358cf-8abc-44e1-a86f-45a254eda159 100 GiB none gzip-9
oxp_d242a551-c4bf-472c-a155-b28b55f1a089/crypt/debug bd779637-f1ff-4efd-ab7c-a2560bf4b2c5 100 GiB none gzip-9
oxp_f4e65604-4406-47f6-a250-a23c8bfd03f8/crypt/debug d90c34cc-f1a8-4e80-a35c-3dfa114cf296 100 GiB none gzip-9
oxp_fbb0a63a-abef-4d79-b2c5-bbdfaf501319/crypt/debug edc979e5-eebd-4f5a-bafa-2d9f91a0fe0a 100 GiB none gzip-9
omicron zones at generation 5:
---------------------------------------------------------------------------------------------
zone type zone id disposition underlay IP
---------------------------------------------------------------------------------------------
cockroach_db bed54974-27d1-4425-bae1-8eddde89dea7 in service fd00:1122:3344:104::3
cockroach_db f98990d5-2f50-4c7a-b816-bbdb807c4712 in service fd00:1122:3344:104::4
crucible 0ffe7a0a-7167-4ecd-ac54-7f8b057a92f0 in service fd00:1122:3344:104::7
crucible 2ddda7b9-e995-4a83-a944-5c9d8c25a76d in service fd00:1122:3344:104::e
crucible 41fc8d59-c462-40c8-bec2-d3d4ae1eb928 in service fd00:1122:3344:104::10
crucible 507e77a0-3027-4d37-81e9-80e5db1f0ae8 in service fd00:1122:3344:104::c
crucible 7b3cf3b6-9fbf-4309-9920-77715ca37b9a in service fd00:1122:3344:104::a
crucible b42afe67-c0f6-4a09-b323-08755282e497 in service fd00:1122:3344:104::8
crucible c5e350f6-15f8-4754-a478-20992d0a0b46 in service fd00:1122:3344:104::d
crucible d5784af6-bcd4-4ace-b399-baad8557136d in service fd00:1122:3344:104::f
crucible ef1a78c5-1afc-4a02-a7b3-5eaab530e6f3 in service fd00:1122:3344:104::b
crucible f33d73b0-0838-408e-85e0-847a73a4b9a3 in service fd00:1122:3344:104::9
crucible_pantry 80d2093d-6d34-4a18-8ee1-964c60e8b8d3 in service fd00:1122:3344:104::6
internal_ntp 6dfeb901-8409-43e3-84ff-86aebe7e0d8a in service fd00:1122:3344:104::11
nexus f19832c3-8f7c-4cd6-bcbd-dc0ef888e11a in service fd00:1122:3344:104::5
sled d2c7dd57-6bd4-4d08-be16-ff770bd0e125 (active):
physical disks at generation 2:
-----------------------------------
vendor model serial
-----------------------------------
1b96 WUS4C6432DSP3X3 A079DDFD
1b96 WUS4C6432DSP3X3 A079DE08
1b96 WUS4C6432DSP3X3 A079DE11
1b96 WUS4C6432DSP3X3 A079DEAF
1b96 WUS4C6432DSP3X3 A079DF11
1b96 WUS4C6432DSP3X3 A079DFA7
1b96 WUS4C6432DSP3X3 A079DFCA
1b96 WUS4C6432DSP3X3 A079E02E
1b96 WUS4C6432DSP3X3 A079E076
datasets at generation 1:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name dataset uuid quota reservation compression
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crypt/cockroachdb cb5eeef8-a1bb-4680-9717-605f024c5fb9 none none off
oxp_3406fa2c-4fa4-4829-b0b3-d44ced78c13d/crucible bf25c418-172e-4de3-b7e2-40459b7c694a none none off
oxp_7e8f608c-8aaf-4581-b1dc-ae6040c24495/crucible 20b2b999-a8c6-4826-8f1b-751f42dc50b2 none none off
oxp_8b6f5a91-d38a-4a34-8fbe-2acddae5f44d/crucible a4fc389d-7db1-4452-bb35-6d1b4b4415ce none none off
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crucible 2f566a43-31bd-498e-88d4-8f73caf7f3b8 none none off
oxp_930836ca-4da6-4e20-93d8-655f0e460e38/crucible 35897692-d90d-4a64-bed2-469be876c099 none none off
oxp_9783b3b0-b1bb-4672-a3e8-00c4fac39016/crucible 7ce962ee-b862-4c7b-99b9-473e6f58152a none none off
oxp_d1007711-8a4a-4ace-98f0-026c4f465acb/crucible 34d27e22-e9d4-4e90-8309-ef388f389962 none none off
oxp_de407fee-9d3c-4f4f-a574-1ac01cafff20/crucible 7263271b-b9cb-4b89-b569-c6d929e30403 none none off
oxp_ecff59fb-f3d5-4313-87f4-c1397f808842/crucible 52bce46c-5530-48d4-9aeb-27d1c4664396 none none off
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crypt/internal_dns 96826543-57f1-4839-9930-3089efdb0f32 none none off
oxp_3406fa2c-4fa4-4829-b0b3-d44ced78c13d/crypt/zone 75bc5669-9c9d-43fd-a79f-d43dba9313e6 none none off
oxp_7e8f608c-8aaf-4581-b1dc-ae6040c24495/crypt/zone b53407f0-cc97-49ed-b3cb-09fa1e4902f1 none none off
oxp_8b6f5a91-d38a-4a34-8fbe-2acddae5f44d/crypt/zone d3428dec-9e25-4c51-8a90-3b2e59fd3071 none none off
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crypt/zone 6babf8f6-817f-47ed-a8b1-17b9135505dc none none off
oxp_930836ca-4da6-4e20-93d8-655f0e460e38/crypt/zone 61d5fd22-0a41-488f-99f3-2736fdb5f03a none none off
oxp_9783b3b0-b1bb-4672-a3e8-00c4fac39016/crypt/zone 286cf4c5-2576-4e1e-aa03-bd5667e7e96a none none off
oxp_d1007711-8a4a-4ace-98f0-026c4f465acb/crypt/zone 48016644-5b9e-4e7e-913e-ebf4c73ccd99 none none off
oxp_de407fee-9d3c-4f4f-a574-1ac01cafff20/crypt/zone ffd4051a-0c4c-498f-8b73-aec6a8e33dce none none off
oxp_ecff59fb-f3d5-4313-87f4-c1397f808842/crypt/zone 3005cb47-e18c-4643-afb9-fd4a753b90ab none none off
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crypt/zone/oxz_cockroachdb_02a8f7ed-d929-4d41-8b18-b115803dbe69 7a568789-ad9e-4a85-a2c2-ea1f8d720e09 none none off
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crypt/zone/oxz_crucible_0229b1c6-9ed5-464f-8119-3019c175a465 271f3430-93c3-419d-aab6-cb544edcaccd none none off
oxp_3406fa2c-4fa4-4829-b0b3-d44ced78c13d/crypt/zone/oxz_crucible_34be4187-ed12-43c8-8a0a-77f2f69baa70 f6db5353-d95d-4f0f-a701-93988a4f0705 none none off
oxp_d1007711-8a4a-4ace-98f0-026c4f465acb/crypt/zone/oxz_crucible_478cb59f-9b6f-4ce0-9317-8ceb366437eb ff5c353c-16da-44f9-bade-50fd8ae31340 none none off
oxp_8b6f5a91-d38a-4a34-8fbe-2acddae5f44d/crypt/zone/oxz_crucible_7ed56b4e-71cd-4832-8d7f-5a02eb2e21a4 cc16b6b7-3b3c-4c8a-a7b2-e88f36db7920 none none off
oxp_de407fee-9d3c-4f4f-a574-1ac01cafff20/crypt/zone/oxz_crucible_90a45278-f307-4898-b91b-077d993b9d72 852541f7-de31-44c1-a644-1d7bf9cf0bb9 none none off
oxp_930836ca-4da6-4e20-93d8-655f0e460e38/crypt/zone/oxz_crucible_9a055248-0595-42dd-b21b-311c72f0578e 8b9af9d5-32cf-46ae-91a3-1db20f5e7139 none none off
oxp_7e8f608c-8aaf-4581-b1dc-ae6040c24495/crypt/zone/oxz_crucible_b030ae01-58c0-4241-ba96-e6b084a8c65a bf8be849-d67c-49ed-ba19-4203d7cab4df none none off
oxp_ecff59fb-f3d5-4313-87f4-c1397f808842/crypt/zone/oxz_crucible_b650dfc3-f06b-41bb-83a8-36bef6b38a06 94685200-fcc2-40b4-a202-e9dd7dbbfc8b none none off
oxp_9783b3b0-b1bb-4672-a3e8-00c4fac39016/crypt/zone/oxz_crucible_e01b9421-472d-4e81-bf88-07ecc1a30043 38813bcf-1ccc-49d0-a04b-26b131b52ab8 none none off
oxp_ecff59fb-f3d5-4313-87f4-c1397f808842/crypt/zone/oxz_crucible_pantry_677bbef8-462e-4241-a032-eacedb409822 a26dfdd4-9a9e-4782-b2df-3d2fd7616e19 none none off
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crypt/zone/oxz_internal_dns_253f67c2-a81a-432a-b09d-7866f8f483f0 af2a583e-ded9-43d3-8116-dda5cce64462 none none off
oxp_8b6f5a91-d38a-4a34-8fbe-2acddae5f44d/crypt/zone/oxz_nexus_53d19ab4-d971-464f-b35d-96a491caaea0 96537c7b-8080-4266-ab01-d7c9055ed472 none none off
oxp_d1007711-8a4a-4ace-98f0-026c4f465acb/crypt/zone/oxz_ntp_12ecb3d8-43ec-4bff-84f3-763dc7a9fc37 d91a6cb6-2c63-4d6f-9452-4d7107afd9d9 none none off
oxp_3406fa2c-4fa4-4829-b0b3-d44ced78c13d/crypt/debug a4600d7b-7f70-449b-bcf7-1dc463d0ab12 100 GiB none gzip-9
oxp_7e8f608c-8aaf-4581-b1dc-ae6040c24495/crypt/debug f048e3a6-513f-4e35-b304-068a70ca1ff2 100 GiB none gzip-9
oxp_8b6f5a91-d38a-4a34-8fbe-2acddae5f44d/crypt/debug 813718e6-2414-4ba4-be17-ea1acfd08dae 100 GiB none gzip-9
oxp_8d18b838-133c-4a5a-9028-3b1d392edcfd/crypt/debug 90f9d72f-00a3-4e61-a7db-013a07b9b0cc 100 GiB none gzip-9
oxp_930836ca-4da6-4e20-93d8-655f0e460e38/crypt/debug 977a0b4e-dfa6-43c2-bafa-1b3f3dd707ce 100 GiB none gzip-9
oxp_9783b3b0-b1bb-4672-a3e8-00c4fac39016/crypt/debug 4febb115-ecb4-43bb-8f31-6069c1ab56aa 100 GiB none gzip-9
oxp_d1007711-8a4a-4ace-98f0-026c4f465acb/crypt/debug dfec11a6-96f3-4fc9-a121-89b8fe78d83a 100 GiB none gzip-9
oxp_de407fee-9d3c-4f4f-a574-1ac01cafff20/crypt/debug c316788d-a078-4dbd-869f-7dab589dd469 100 GiB none gzip-9
oxp_ecff59fb-f3d5-4313-87f4-c1397f808842/crypt/debug 828e4385-43c6-4b5b-ab0d-8658be9e94e4 100 GiB none gzip-9
omicron zones at generation 5:
--------------------------------------------------------------------------------------------
zone type zone id disposition underlay IP
--------------------------------------------------------------------------------------------
cockroach_db 02a8f7ed-d929-4d41-8b18-b115803dbe69 in service fd00:1122:3344:103::3
crucible 0229b1c6-9ed5-464f-8119-3019c175a465 in service fd00:1122:3344:103::6
crucible 34be4187-ed12-43c8-8a0a-77f2f69baa70 in service fd00:1122:3344:103::c
crucible 478cb59f-9b6f-4ce0-9317-8ceb366437eb in service fd00:1122:3344:103::b
crucible 7ed56b4e-71cd-4832-8d7f-5a02eb2e21a4 in service fd00:1122:3344:103::a
crucible 90a45278-f307-4898-b91b-077d993b9d72 in service fd00:1122:3344:103::d
crucible 9a055248-0595-42dd-b21b-311c72f0578e in service fd00:1122:3344:103::e
crucible b030ae01-58c0-4241-ba96-e6b084a8c65a in service fd00:1122:3344:103::8
crucible b650dfc3-f06b-41bb-83a8-36bef6b38a06 in service fd00:1122:3344:103::9
crucible e01b9421-472d-4e81-bf88-07ecc1a30043 in service fd00:1122:3344:103::7
crucible_pantry 677bbef8-462e-4241-a032-eacedb409822 in service fd00:1122:3344:103::5
internal_dns 253f67c2-a81a-432a-b09d-7866f8f483f0 in service fd00:1122:3344:3::1
internal_ntp 12ecb3d8-43ec-4bff-84f3-763dc7a9fc37 in service fd00:1122:3344:103::f
nexus 53d19ab4-d971-464f-b35d-96a491caaea0 in service fd00:1122:3344:103::4
MODIFIED SLEDS:
sled 4cf9afc2-a1ef-4015-a57f-8c3e110e6641 (active):
physical disks generation 2 -> 3:
-----------------------------------
vendor model serial
-----------------------------------
1b96 WUS4C6432DSP3X3 A079DDCC
1b96 WUS4C6432DSP3X3 A079DE55
1b96 WUS4C6432DSP3X3 A079DF02
1b96 WUS4C6432DSP3X3 A079DF08
1b96 WUS4C6432DSP3X3 A079E3AD
1b96 WUS4C6432DSP3X3 A079E3D2
1b96 WUS4C6432DSP3X3 A079E567
1b96 WUS4C6432DSP3X3 A079E665
- 1b96 WUS4C6432DSP3X3 A079DDE7
datasets generation 1 -> 2:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name dataset uuid quota reservation compression
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/cockroachdb b34f358f-64a1-4e8b-9981-b042acad0445 none none off
oxp_11b50275-8a5e-482f-a1d0-7485c52ce61c/crucible b723a7ed-556f-4a4d-b00f-f2bfd58dc421 none none off
oxp_1c4e8ca0-1b98-44c7-bdf4-64c28fe011f6/crucible ae9e4e45-864e-430b-b9a9-bdd61ecb3d98 none none off
oxp_3ec452c0-a9a7-486e-9214-7df3329e41ad/crucible cd5aa5bd-dfbf-471c-80fb-f83b25117e3e none none off
oxp_56845ecb-c52c-42af-a8f0-fa5c3fa793e8/crucible 7a2dc70a-297e-46ec-b48f-96a2749b2b8b none none off
oxp_5e2eaa32-1f03-4312-98a7-9a6492b8956e/crucible 1ff8ef91-aa5d-4dfa-924a-e196b6a73eab none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crucible 3411958a-808d-449d-8024-a60e4f473d5d none none off
oxp_afc931ec-f082-4bc3-8223-accd4ff0f1a2/crucible 54721f71-e207-465f-aa9b-c04878d0aaab none none off
oxp_fd44bfe1-c2fd-4f45-9e00-8dec76148395/crucible e2eac5e2-2b29-4271-bd26-317d69b1621c none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/external_dns 4e6556eb-e97c-44f5-b243-49471b675c91 none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/internal_dns 3ef3d212-a565-4559-9052-9e98e540838b none none off
oxp_11b50275-8a5e-482f-a1d0-7485c52ce61c/crypt/zone a0b6aa75-dadb-4e95-98e5-65804a28bd2e none none off
oxp_1c4e8ca0-1b98-44c7-bdf4-64c28fe011f6/crypt/zone db7e1011-f4f6-4430-a924-b795d124e6fd none none off
oxp_3ec452c0-a9a7-486e-9214-7df3329e41ad/crypt/zone 146fc963-8c69-4fb9-8ee6-280b962e33d3 none none off
oxp_56845ecb-c52c-42af-a8f0-fa5c3fa793e8/crypt/zone 724ae916-7fba-4bae-93ba-0fbe1c3ea8f4 none none off
oxp_5e2eaa32-1f03-4312-98a7-9a6492b8956e/crypt/zone f84ecc32-e519-48b1-90c6-e091302a7f15 none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/zone 7731ee0d-c131-423c-b227-4d8db6c1cef1 none none off
oxp_afc931ec-f082-4bc3-8223-accd4ff0f1a2/crypt/zone 8df75f4f-f1d1-47ab-a832-4c840357a33d none none off
oxp_fd44bfe1-c2fd-4f45-9e00-8dec76148395/crypt/zone 8b56aa3d-a8bd-4f6e-a074-321203f07126 none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/zone/oxz_cockroachdb_bb3bc34a-45f9-400d-ab0c-38b526ff0d78 dabb3474-fa56-4b9f-b019-b75d2ae51759 none none off
oxp_56845ecb-c52c-42af-a8f0-fa5c3fa793e8/crypt/zone/oxz_crucible_1dcc1fa5-0061-418a-9147-a7557dd5b80f 6a764afc-a271-491a-ab81-109e8545c4e0 none none off
oxp_1c4e8ca0-1b98-44c7-bdf4-64c28fe011f6/crypt/zone/oxz_crucible_262b4cc3-9215-4993-97d8-e6b9dd98f337 824482a6-e55e-4259-85b0-93b553e3db35 none none off
oxp_fd44bfe1-c2fd-4f45-9e00-8dec76148395/crypt/zone/oxz_crucible_53d33856-60cd-4921-bc77-095d2419d8b8 b83365c2-c516-45e7-bf11-de4463cb339b none none off
oxp_3ec452c0-a9a7-486e-9214-7df3329e41ad/crypt/zone/oxz_crucible_62aeb960-9dba-457c-8e0f-f866246b4ab1 e1582c71-c596-4c53-b83f-b55bd9bd21bf none none off
oxp_11b50275-8a5e-482f-a1d0-7485c52ce61c/crypt/zone/oxz_crucible_6cdf7f43-e984-449b-aa21-cf50eff7eca5 d9362d04-21ae-40b0-aa27-9183bc580da9 none none off
oxp_5e2eaa32-1f03-4312-98a7-9a6492b8956e/crypt/zone/oxz_crucible_72e95a3e-cf41-4a07-a9d6-127c17714ddb 86eebe3a-6514-4a05-afe7-4eee7b586c4b none none off
oxp_afc931ec-f082-4bc3-8223-accd4ff0f1a2/crypt/zone/oxz_crucible_9ce415a2-a533-42cf-80ff-867a04394a5d 4d9763da-ba3e-4119-b718-2e6b72d1ece4 none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/zone/oxz_crucible_9e8f6eab-b8f0-451d-b7dc-db002f8f16bc 53e11f09-57b8-44c1-a27e-95e837fa6eb9 none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/zone/oxz_external_dns_e15d6e9b-aea5-4a82-aa3d-f1f19a27d86f a7d41777-aa8a-4679-ac2c-c56b803848b7 none none off
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/zone/oxz_internal_dns_e45e8a6b-7c5d-4d8c-8a6f-9803bf6d3cfd 42c4b905-16b2-407b-93c5-f36b5cdd7616 none none off
oxp_1c4e8ca0-1b98-44c7-bdf4-64c28fe011f6/crypt/zone/oxz_ntp_836b8090-fcc0-4ba8-a03d-8a281a03e30c acfb0a75-98e1-4b7c-aad5-2fc592ce9080 none none off
oxp_56845ecb-c52c-42af-a8f0-fa5c3fa793e8/crypt/zone/oxz_oximeter_ef7bec0e-a5b8-4e2c-929a-520f26622ab8 643a61f2-d59b-40bc-b17e-702008dd05f0 none none off
oxp_11b50275-8a5e-482f-a1d0-7485c52ce61c/crypt/debug d1eb4ab3-f9e8-43ef-a794-6476219db13a 100 GiB none gzip-9
oxp_1c4e8ca0-1b98-44c7-bdf4-64c28fe011f6/crypt/debug dedfb791-776b-423c-8bc4-49282dd85629 100 GiB none gzip-9
oxp_3ec452c0-a9a7-486e-9214-7df3329e41ad/crypt/debug 37df357a-63be-463c-8741-fc99a3e608d2 100 GiB none gzip-9
oxp_56845ecb-c52c-42af-a8f0-fa5c3fa793e8/crypt/debug 54bdbfe7-3b6a-47a6-b683-3d1d726a2c26 100 GiB none gzip-9
oxp_5e2eaa32-1f03-4312-98a7-9a6492b8956e/crypt/debug 4fb8b1ef-633e-439a-a022-3c48c485754b 100 GiB none gzip-9
oxp_6f6d99e5-5410-4852-adf3-4783d729935b/crypt/debug 52105215-ea2d-499b-a560-715d679f03c4 100 GiB none gzip-9
oxp_afc931ec-f082-4bc3-8223-accd4ff0f1a2/crypt/debug e6e09ce7-cbf5-4365-8cd0-abff8bf60a4e 100 GiB none gzip-9
oxp_fd44bfe1-c2fd-4f45-9e00-8dec76148395/crypt/debug a2608511-8219-41a4-bcca-23691a11655c 100 GiB none gzip-9
* oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6/crucible 27dc68ba-59b8-4ba1-af83-b47eba809317 none none off
* oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6/crypt/zone 232e4106-48f1-4682-b9a3-ebd15d8e7fec none none off
* oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6/crypt/zone/oxz_crucible_5e0d1ddd-6f67-490a-bcdb-d5bb73873b9b 12dde9f8-0f07-4cb3-be2b-2717e5921bc8 none none off
* oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6/crypt/zone/oxz_crucible_pantry_4899139b-2654-4b65-83ac-1424335ff646 c8415c5d-2573-4eb1-a5b9-8946c2293462 none none off
* oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6/crypt/debug e7d505a1-a600-4582-bccb-74db91622978 100 GiB none gzip-9
omicron zones generation 5 -> 6:
----------------------------------------------------------------------------------------------
zone type zone id disposition underlay IP
----------------------------------------------------------------------------------------------
boundary_ntp 836b8090-fcc0-4ba8-a03d-8a281a03e30c in service fd00:1122:3344:101::10
cockroach_db bb3bc34a-45f9-400d-ab0c-38b526ff0d78 in service fd00:1122:3344:101::3
crucible 1dcc1fa5-0061-418a-9147-a7557dd5b80f in service fd00:1122:3344:101::a
crucible 262b4cc3-9215-4993-97d8-e6b9dd98f337 in service fd00:1122:3344:101::e
crucible 53d33856-60cd-4921-bc77-095d2419d8b8 in service fd00:1122:3344:101::f
crucible 62aeb960-9dba-457c-8e0f-f866246b4ab1 in service fd00:1122:3344:101::c
crucible 6cdf7f43-e984-449b-aa21-cf50eff7eca5 in service fd00:1122:3344:101::d
crucible 72e95a3e-cf41-4a07-a9d6-127c17714ddb in service fd00:1122:3344:101::9
crucible 9ce415a2-a533-42cf-80ff-867a04394a5d in service fd00:1122:3344:101::8
crucible 9e8f6eab-b8f0-451d-b7dc-db002f8f16bc in service fd00:1122:3344:101::7
external_dns e15d6e9b-aea5-4a82-aa3d-f1f19a27d86f in service fd00:1122:3344:101::4
internal_dns e45e8a6b-7c5d-4d8c-8a6f-9803bf6d3cfd in service fd00:1122:3344:1::1
oximeter ef7bec0e-a5b8-4e2c-929a-520f26622ab8 in service fd00:1122:3344:101::5
* crucible 5e0d1ddd-6f67-490a-bcdb-d5bb73873b9b - in service fd00:1122:3344:101::b
└─ + expunged
* crucible_pantry 4899139b-2654-4b65-83ac-1424335ff646 - in service fd00:1122:3344:101::6
└─ + expunged
sled 9be81ab1-f82d-4ee9-bc0d-9466448f8fa3 (active):
physical disks at generation 2:
-----------------------------------
vendor model serial
-----------------------------------
1b96 WUS4C6432DSP3X3 A079DE84
1b96 WUS4C6432DSP3X3 A079DEE9
1b96 WUS4C6432DSP3X3 A079DF1E
1b96 WUS4C6432DSP3X3 A079DFBF
1b96 WUS4C6432DSP3X3 A079E184
1b96 WUS4C6432DSP3X3 A079E342
1b96 WUS4C6432DSP3X3 A079E35A
1b96 WUS4C6432DSP3X3 A079E3AE
1b96 WUS4C6432DSP3X3 A079E708
1b96 WUS4C6432DSP3X3 A084A7EA
datasets generation 1 -> 2:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name dataset uuid quota reservation compression
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/cockroachdb 4d926f79-e5bd-439c-8fb4-6f9df021fdfa none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crucible 238a7357-91fe-4d11-9523-a127dd08b420 none none off
oxp_72976e99-7fec-4f96-adf9-36b502ce33e1/crucible b975f11b-81b4-4be9-88d4-5f8741617a7c none none off
oxp_74aff542-7528-4aee-9042-446911e765ec/crucible 848eb0f5-4ed6-4c0a-9b69-553174057609 none none off
oxp_81afa157-235d-4917-a061-f146f9085ff0/crucible 7ff0a48b-aa28-448b-b9e4-3f264af05fbe none none off
oxp_d4889c6a-4210-446b-9135-8805a956f395/crucible c66204dd-da58-4c13-a11e-5fdf10fd027c none none off
oxp_dc129a48-5eee-4fe0-bd2f-0cb2bc85ab68/crucible 4ce41e9f-f84c-4921-b9de-0ac6a249f8ad none none off
oxp_e6e7a979-09ad-4f5e-8d44-295e80020cc7/crucible 5f9010a6-52cb-4904-80a9-0e5a587ae922 none none off
oxp_e844a39c-189e-4a5d-b960-97cee29dc1cb/crucible 02871ff6-cd96-4dee-960e-2c4a9ba899d2 none none off
oxp_ed777c93-c036-4a45-95e0-9ad50cb25b17/crucible 9418f16d-5234-4c27-b1a6-79a8d33d6f9e none none off
oxp_fc3f7898-1d3d-44ad-a06a-9e2e303e48ac/crucible dbfc301a-fcc2-43c2-8a0d-5104c647f243 none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/clickhouse 69d366c0-7bd3-4755-bc24-c20a0b2d9aa9 none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/internal_dns 79b30bca-4a39-41f4-aa8c-a1ba0601cc0a none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/zone ba0c8903-cda6-4e7b-b1c0-14a646560fdc none none off
oxp_72976e99-7fec-4f96-adf9-36b502ce33e1/crypt/zone bf63e7ab-bd7a-4417-bb11-c86f670fcc3f none none off
oxp_74aff542-7528-4aee-9042-446911e765ec/crypt/zone 8e544652-8aaa-404e-a269-2cb731c776e0 none none off
oxp_81afa157-235d-4917-a061-f146f9085ff0/crypt/zone 31a518dd-93b6-4231-8a53-9e1c2b836ae3 none none off
oxp_d4889c6a-4210-446b-9135-8805a956f395/crypt/zone ef5dfa1b-1ffb-483d-add8-75b0ccd09982 none none off
oxp_dc129a48-5eee-4fe0-bd2f-0cb2bc85ab68/crypt/zone 57db3ad8-b3ea-4553-b031-add317d3b582 none none off
oxp_e6e7a979-09ad-4f5e-8d44-295e80020cc7/crypt/zone 0c4cdd08-1225-4224-bdcc-c0be23af939d none none off
oxp_e844a39c-189e-4a5d-b960-97cee29dc1cb/crypt/zone b7343ac5-a4c2-4fc5-b49b-7878dcd1149a none none off
oxp_ed777c93-c036-4a45-95e0-9ad50cb25b17/crypt/zone 3c17688b-e089-4fb8-9705-49b55d1be969 none none off
oxp_fc3f7898-1d3d-44ad-a06a-9e2e303e48ac/crypt/zone b33b7398-ef7d-44d3-badc-4e49cecb7e3d none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/zone/oxz_clickhouse_e12e7fdf-988f-43fa-bd7a-0ba1561f19a9 6d13c437-78fa-4f98-a2a0-4c0e3c74248d none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/zone/oxz_cockroachdb_c7da8ea9-74c6-40db-a5a5-d28a3919fe18 bcc6dd1d-3ce0-4768-8bef-befb4d24b396 none none off
oxp_ed777c93-c036-4a45-95e0-9ad50cb25b17/crypt/zone/oxz_crucible_1614d6f8-6edb-408b-9920-e5767342850c 7048f112-22cb-42dc-980d-12468bdd3558 none none off
oxp_dc129a48-5eee-4fe0-bd2f-0cb2bc85ab68/crypt/zone/oxz_crucible_17b3698c-732d-4036-9432-ca704472ca04 8bba2709-c273-4475-86ce-c1e273a05ccd none none off
oxp_74aff542-7528-4aee-9042-446911e765ec/crypt/zone/oxz_crucible_335e6304-bfdb-4a02-b352-8bada9638ed8 aac8e907-1762-40b2-b880-392beb403000 none none off
oxp_81afa157-235d-4917-a061-f146f9085ff0/crypt/zone/oxz_crucible_70c42b78-49dd-4446-b05e-4451350f79e0 6cc06ece-aea4-480e-9a49-1a15793ec1d0 none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/zone/oxz_crucible_acf5d72e-8661-4d19-815a-bf1b117566db a0a5f692-2c75-4fb9-a160-c98d8df10067 none none off
oxp_e844a39c-189e-4a5d-b960-97cee29dc1cb/crypt/zone/oxz_crucible_bd65be16-7828-4a7f-8f38-8580abb01f33 396dba5a-58b7-459e-a87b-abecec646a11 none none off
oxp_d4889c6a-4210-446b-9135-8805a956f395/crypt/zone/oxz_crucible_dc777f78-533d-4b63-b788-bcdd520be8f3 3a98117d-a62f-42f8-883d-d326859fc196 none none off
oxp_72976e99-7fec-4f96-adf9-36b502ce33e1/crypt/zone/oxz_crucible_f01e2880-29d2-4afe-9326-c90189fb9101 813b3384-df91-43ad-99f1-bf2a1735ca12 none none off
oxp_fc3f7898-1d3d-44ad-a06a-9e2e303e48ac/crypt/zone/oxz_crucible_f6c522a1-205d-498e-beb0-582e1cd39582 c1c0557a-4a48-44b3-b9b1-126a42c36481 none none off
oxp_e6e7a979-09ad-4f5e-8d44-295e80020cc7/crypt/zone/oxz_crucible_fdd6bea9-5f87-46ce-8ba8-181e36bb629e 4db18604-2d3a-4af5-b364-0a1a156540e9 none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/zone/oxz_internal_dns_72bc2263-330e-404a-a343-3d2d0b26ace7 fe34993c-17c7-45bc-a04a-9aeb3762c48c none none off
oxp_ed777c93-c036-4a45-95e0-9ad50cb25b17/crypt/zone/oxz_nexus_00583164-bf4f-49f9-b221-b052c20a6c0f 5262d4b6-4d6d-44c2-8169-6b6ef7c3ddb9 none none off
oxp_e844a39c-189e-4a5d-b960-97cee29dc1cb/crypt/zone/oxz_ntp_94021e5c-8ca9-497a-9341-54b494a0d6b8 05cfb04b-a886-4ece-bfff-93194700a77f none none off
oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/debug 638d772f-458c-4830-9b43-b63601a4df8e 100 GiB none gzip-9
oxp_72976e99-7fec-4f96-adf9-36b502ce33e1/crypt/debug d8574a0e-5be3-4fb3-8f6e-880224287e89 100 GiB none gzip-9
oxp_74aff542-7528-4aee-9042-446911e765ec/crypt/debug af45918b-0b16-4829-ab4d-a7bd4608e131 100 GiB none gzip-9
oxp_81afa157-235d-4917-a061-f146f9085ff0/crypt/debug 63993b09-c1d4-4330-b6ed-725d8d88f43d 100 GiB none gzip-9
oxp_d4889c6a-4210-446b-9135-8805a956f395/crypt/debug eaa691bc-8beb-40b4-b519-ce0958702700 100 GiB none gzip-9
oxp_dc129a48-5eee-4fe0-bd2f-0cb2bc85ab68/crypt/debug 92ca8d1f-fe3b-47a4-954c-5b25a9268a2b 100 GiB none gzip-9
oxp_e6e7a979-09ad-4f5e-8d44-295e80020cc7/crypt/debug a84300cd-9f01-4cb5-9d24-f77b6b4de54c 100 GiB none gzip-9
oxp_e844a39c-189e-4a5d-b960-97cee29dc1cb/crypt/debug 8d76a1ec-b038-4820-b16e-f8d3bec25803 100 GiB none gzip-9
oxp_ed777c93-c036-4a45-95e0-9ad50cb25b17/crypt/debug 5ddb46ec-26e4-4151-90cf-78b717900452 100 GiB none gzip-9
oxp_fc3f7898-1d3d-44ad-a06a-9e2e303e48ac/crypt/debug 03f7c2ad-856d-432d-8104-0eddb0e2c32a 100 GiB none gzip-9
+ oxp_2d019d82-168a-4b35-96d9-c81007fcb9fe/crypt/zone/oxz_crucible_pantry_35e9cd6e-f3a3-42c0-bb92-f64627606bde 66108ef3-9ea6-4745-b9da-3a1b3c69f2ad none none off
omicron zones generation 5 -> 6:
---------------------------------------------------------------------------------------------
zone type zone id disposition underlay IP
---------------------------------------------------------------------------------------------
boundary_ntp 94021e5c-8ca9-497a-9341-54b494a0d6b8 in service fd00:1122:3344:102::10
clickhouse e12e7fdf-988f-43fa-bd7a-0ba1561f19a9 in service fd00:1122:3344:102::5
cockroach_db c7da8ea9-74c6-40db-a5a5-d28a3919fe18 in service fd00:1122:3344:102::3
crucible 1614d6f8-6edb-408b-9920-e5767342850c in service fd00:1122:3344:102::d
crucible 17b3698c-732d-4036-9432-ca704472ca04 in service fd00:1122:3344:102::b
crucible 335e6304-bfdb-4a02-b352-8bada9638ed8 in service fd00:1122:3344:102::8
crucible 70c42b78-49dd-4446-b05e-4451350f79e0 in service fd00:1122:3344:102::c
crucible acf5d72e-8661-4d19-815a-bf1b117566db in service fd00:1122:3344:102::6
crucible bd65be16-7828-4a7f-8f38-8580abb01f33 in service fd00:1122:3344:102::f
crucible dc777f78-533d-4b63-b788-bcdd520be8f3 in service fd00:1122:3344:102::9
crucible f01e2880-29d2-4afe-9326-c90189fb9101 in service fd00:1122:3344:102::a
crucible f6c522a1-205d-498e-beb0-582e1cd39582 in service fd00:1122:3344:102::7
crucible fdd6bea9-5f87-46ce-8ba8-181e36bb629e in service fd00:1122:3344:102::e
internal_dns 72bc2263-330e-404a-a343-3d2d0b26ace7 in service fd00:1122:3344:2::1
nexus 00583164-bf4f-49f9-b221-b052c20a6c0f in service fd00:1122:3344:102::4
+ crucible_pantry 35e9cd6e-f3a3-42c0-bb92-f64627606bde in service fd00:1122:3344:102::21
COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: d4d87aa2ad877a4cc2fddd0573952362739110de (unchanged)
cluster.preserve_downgrade_option: "22.1" (unchanged)
METADATA:
internal DNS version: 1 (unchanged)
external DNS version: 2 (unchanged)
root@oxz_switch0:~# omdb -w nexus blueprints target set 4645bedc-1160-4100-8ea7-8082e3b18954 enabled
set target blueprint to 4645bedc-1160-4100-8ea7-8082e3b18954
root@oxz_switch0:~# omdb nexus background-tasks show blueprint_executor
task: "blueprint_executor"
configured period: every 1m
currently executing: no
last completed activation: iter 55, triggered by a periodic timer firing
started at 2025-02-06T20:01:49.416Z (32s ago) and ran for 5053ms
target blueprint: 4645bedc-1160-4100-8ea7-8082e3b18954
execution: enabled
status: failed at: Deploy datasets (step 5/16)
error: step failed: Deploy datasets
caused by: failure deploying datasets: [DatasetManagementStatus { dataset_name: DatasetName { kind: TransientZone { name: "oxz_crucible_5e0d1ddd-6f67-490a-bcdb-d5bb73873b9b" }, pool_name: ZpoolName { id: 3ef8333c-94a5-4650-a62a-9e440a1eabc6 (zpool), kind: External } }, err: Some("Zpool Not Found: oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6") }, DatasetManagementStatus { dataset_name: DatasetName { kind: TransientZoneRoot, pool_name: ZpoolName { id: 3ef8333c-94a5-4650-a62a-9e440a1eabc6 (zpool), kind: External } }, err: Some("Zpool Not Found: oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6") }, DatasetManagementStatus { dataset_name: DatasetName { kind: TransientZone { name: "oxz_crucible_pantry_4899139b-2654-4b65-83ac-1424335ff646" }, pool_name: ZpoolName { id: 3ef8333c-94a5-4650-a62a-9e440a1eabc6 (zpool), kind: External } }, err: Some("Zpool Not Found: oxp_3ef8333c-94a5-4650-a62a-9e440a1eabc6") }]