update to dpd-api results in version failure from sled agent

[rcgoodfellow]

We recently had several version bumps in the Dendrite dpd API that landed in Omicron. Omicron treats dpd as a server-side versioned API. That is true for nexus. However, it is not generally true for sled agent. Sled agents can run ahead of dpd which can mean version errors when they call dpd. For critical setup functions like establishing NAT for NTP zones, this can torpedo an update, where we essentially find ourselves in problem 3 described here

• sled-agent's setup of OPTE port NAT entries is unreliable #9700

[davepacheco]

Failure mode

The failure mode for this was that a dogfood update got stuck (at the point @rcgoodfellow mentioned) with this error:

00:03:25.059Z WARN SledAgent (ServiceManager): failed to create NAT entry for service
    error = DpdError(Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "de90ac5b-8359-4872-8381-32db93d70585", "content-length": "122", "date": "Thu, 22 Jan 2026 01:32:30 GMT"}; value: Error { error_code: None, message: "server does not support this API version: 4.0.0", request_id: "de90ac5b-8359-4872-8381-32db93d70585" })
    file = sled-agent/src/services.rs:1222
    zone_type = boundary_ntp

Aside from it being dogfood and not a customer system, this is basically the nightmare scenario for upgrade: we've introduced an incompatible API change that was undetected by all the tooling and it led to a hang of the update itself.

Recovery

For any system that hits this problem, it should be possible to move past it by using the official procedures for MUPdating all sleds to the same version that's being updated to.

Impact

As far as I understand, this problem would break self-service update from R17 (or one of its patch releases) to anything after when dpd v2 was introduced (back in October, but will not be released until R18), so this is an R18 blocker.

Analysis

In terms of the versioning: we determined (incorrectly) in #7848 that dpd could be server-side-versioned because there is no cycle among server-side-versioned components introduced by doing that. The thinking was that that means there's an update sequence that ensures that dpd is always updated before its clients. That would be true if we update individual components like this -- we'd literally just update dpd before sled agent. But that's not how it works: components are grouped into deployment units and the update sequence is in terms of deployment units, not these components. And there is no sequence of deployment unit updates that ensures that dpd is always updated before its clients.

Another way to think of this mistake: we did look at the graph of deployment units in determining the update sequence (see RFD 565 section 6 "Updates sequence", but we failed to account for the fact that there's an edge from "Host OS" to itself. That edge says gateway-client, but having just been poking around in ls-apis, there are actually several edges there and only the latest ever gets shown.

I'll file a separate issue for the tooling bug that should have caught this.

A separate question is: why didn't we see this when we updated dogfood to the version where dpd v2 was introduced? My guess is that either that particular update was a MUPdate or else we updated to a middle TUF repo in which dpd supported v2 but sled agent wasn't yet using it. This is possible because Omicron generally lags Dendrite a bit. I have not confirmed this.

Proposed fix

Change sled agent to use dpd-client v1. It does not appear to use any of the changes in newer versions so this is a source-compatible change. v1 was present in R17 and still supported in dpd in R18 (and it has to be per the rules of server-side-versioned APIs), so this should be an easy, low-risk fix.