Add Nexus sagas for external subnet attach / detach by bnaecker · Pull Request #9779 · oxidecomputer/omicron

bnaecker · 2026-02-03T00:55:35Z

Adds a subnet attach and detach saga in Nexus, modeled after the existing floating IP attachment sagas.
Update the common code for sagas to include passing or removing the attached subnets to Dendrite and / or OPTE. This is to pick up those changes during the existing instance sagas, e.g. instance update.
Fixes Saga for attaching external subnets to instances #9685

- Add APIs to the sled agent for attaching and detaching either a single subnet on an instance, or setting / clearing the entire set for an instance. - Add list of attached subnets in the instance-creation request body, and fill that in from Nexus with the (currently-empty) set of attached subnets for the target instnace. - Plumb attachment requests all the way through the sled-agent internals to the new APIs in OPTE. - Add mapping of attached subnets per-instance to the simulated sled agent for testing. - Fixes #9702

- Adds a subnet attach and detach saga in Nexus, modeled after the existing floating IP attachment sagas. - Update the common code for sagas to include passing or removing the attached subnets to Dendrite and / or OPTE. This is to pick up those changes during the existing instance sagas, e.g. instance update. - Fixes #9685

bnaecker · 2026-02-03T01:01:56Z

Stacked on #9778

bnaecker · 2026-02-03T05:43:24Z

So the failure on the helios-deploy job looks like a race. We can see that it attempts to detect the switch zone as being up by curling against the Dendrite API server. It last does that at 2026-02-03T02:20:45.222Z and reports a failure. When collecting evidence, we try to login to the switch zone for various things, such as collecting the link state. There we see this:

2026-02-03T02:20:45.634Z	+ pfexec zlogin oxz_switch /opt/oxide/dendrite/bin/swadm link ls
2026-02-03T02:20:45.634Z	Error: failed to list all links
2026-02-03T02:20:45.634Z	
2026-02-03T02:20:45.634Z	Caused by:
2026-02-03T02:20:45.634Z	    0: Communication Error: error sending request for url (http://localhost:12224/links)
2026-02-03T02:20:45.634Z	    1: error sending request for url (http://localhost:12224/links)
2026-02-03T02:20:45.634Z	    2: client error (Connect)
2026-02-03T02:20:45.644Z	    3: tcp connect error
2026-02-03T02:20:45.644Z	    4: Connection refused (os error 146)

That means swadm can't even talk to Dendrite over localhost! But looking in the saved Dendrite logs, we see this at the top:

2026-02-03T02:20:46.666Z	INFO	dpd: dpd config: Config

So it seems like Dendrite had not started by the time we started, or even failed, the check for the switch zone being up. I'm guessing this is fallout from #9767, and we were succeeding before by accident. We probably need to increase that timeout, but it would also be good to know why the Dendrite service is taking so long to start here. The SMF log file shows the service was enabled at 02:20:15, but the start method didn't run for another 30s. We might want to look at how the sled-agent initializes this zone, and whether this lag is expected.

nexus/db-model/src/external_subnet.rs

nexus/db-queries/src/db/datastore/external_subnet.rs

nexus/src/app/instance_network.rs

nexus/src/app/sagas/subnet_detach.rs

nexus/src/app/sagas/subnet_attach.rs

- Delete entries from all switches, not just boundary - Keep `rack_id` name for future API compat - Respect the `do_saga` flag

nexus/src/app/instance_network.rs

nexus/db-queries/src/db/datastore/external_subnet.rs

nexus/db-queries/src/db/queries/external_subnet.rs

nexus/db-queries/src/db/datastore/external_subnet.rs

nexus/db-schema/src/schema.rs

FelixMcFelix

Thanks for being patient, mainly docs nits etc. outside of the deletion comment I left earlier.

nexus/db-queries/src/db/datastore/external_subnet.rs

nexus/src/app/sagas/subnet_attach.rs

nexus/src/app/sagas/subnet_detach.rs

- Improved naming of consts and methods - Many comment improvements in subnet {at,de}tach methods, point to locking notes in instance IP attach saga, improve handling of deleted instances during subnet sagas - Add test that detaching instance subnets doesn't delete the subnets.

FelixMcFelix

Fab, thanks for putting this together!

bnaecker · 2026-02-04T22:50:04Z

Going to merge in main, and then set this to automerge. Thanks for the careful reviews everyone!

…sagas

bnaecker added 2 commits February 3, 2026 00:52

bnaecker added this to the 18 milestone Feb 3, 2026

bnaecker mentioned this pull request Feb 3, 2026

Add sled-agent API for managing attached subnets #9778

Merged

rcgoodfellow reviewed Feb 3, 2026

View reviewed changes

bnaecker commented Feb 3, 2026

View reviewed changes

nexus/src/app/sagas/subnet_attach.rs Show resolved Hide resolved

bnaecker requested a review from FelixMcFelix February 4, 2026 04:17

Review feedback

014d94d

- Delete entries from all switches, not just boundary - Keep `rack_id` name for future API compat - Respect the `do_saga` flag

mergeconflict approved these changes Feb 4, 2026

View reviewed changes

Base automatically changed from sled-agent-attached-subnet-api to main February 4, 2026 17:24

FelixMcFelix reviewed Feb 4, 2026

View reviewed changes

bnaecker requested review from FelixMcFelix, mergeconflict and rcgoodfellow February 4, 2026 20:16

FelixMcFelix approved these changes Feb 4, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into subnet-attach-detach-…

08d42c1

…sagas

bnaecker enabled auto-merge (squash) February 4, 2026 23:14

Remove stale sled-agent OpenAPI doc

d3d57f4

bnaecker merged commit 08c0d9b into main Feb 5, 2026
16 checks passed

bnaecker deleted the subnet-attach-detach-sagas branch February 5, 2026 03:09

Conversation

bnaecker commented Feb 3, 2026

Uh oh!

bnaecker commented Feb 3, 2026

Uh oh!

bnaecker commented Feb 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FelixMcFelix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FelixMcFelix left a comment

Choose a reason for hiding this comment

Uh oh!

bnaecker commented Feb 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants