Skip to content

feat(service): apply definitions without a daemon restart - #1119

Merged
edwin-zvs merged 2 commits into
mainfrom
service-live-reload
Aug 1, 2026
Merged

feat(service): apply definitions without a daemon restart#1119
edwin-zvs merged 2 commits into
mainfrom
service-live-reload

Conversation

@edwin-zvs

Copy link
Copy Markdown
Contributor

The problem

Service definitions were loaded once at boot, so every edit returned restart_required: true and did nothing until the daemon restarted. Parts of that were worse than a missing feature:

  • paused was a lie. Consulted only at startup, so pausing a running service left its listener bound and serving.
  • Rotating a credential left the old one working, since the token was snapshotted when the listener bound.
  • Both UIs asserted the restriction in prose rather than reading restart_required, which nothing consumed — ~14 hardcoded strings.

What applies when

Published as shared data (PropagationClass / ServiceField) that the daemon and its clients both read, so the promise beside a field is the promise the daemon keeps:

Change Applies
Channel attachment, port, enabled, credential immediately
Routing rule, paused, approval timeout next request
Instruction, harness, model, cwd, sandbox new sessions

Instruction and model stay next-session deliberately. A harness assembles its system prompt at startup, and resuming one skips its seed prompt (agent.rs, if !persist::is_resume()) so the model isn't charged twice — so respawning re-delivers nothing and there is no honest way to re-instruct a live conversation. The classes say that rather than implying otherwise.

Design

A supervisor task (modeled on remote_supervisor) owns every listener; the registry lives in its stack frame, so reloads serialize by construction. Config is read at the point of use behind RwLock<Arc<ServiceConfig>>, so only a port change touches a socket.

  • All or nothing. A definition that fails to parse leaves the running configuration untouched — which is also what makes watching the config directory safe against a file caught mid-write.
  • Stops complete before starts, awaiting each handle, so two channels can exchange ports (the port frees when the listener drops inside its task, not at cancellation).
  • Connections drain, sockets rebind. Cancelling stops accepting; a request already being served runs to completion, because it has an agent turn behind it.
  • ServiceShared is never rebuilt for a name that already has one — two would race on the same state file and reset the dedup ring, stranding live conversations and admitting duplicate deliveries.
  • Duplicate ports are reported, and the already-serving claim keeps the port.
  • restart_requiredServiceApplyResult reporting the socket work actually performed.

Also fixed in passing: persist_state wrote without tmp+rename, so a crash mid-write lost every routed session mapping.

Verification

Live daemon, every change made by hand-editing TOML, no restart:

rotate credential  old token -> 401,  new token -> 202
routing rule       per-event now accepted without session_key (was 400)
pause              connection refused, port released
resume + move port old port free, new port -> 202
malformed TOML     still serving during bad parse, recovers when fixed
service.delete     returns null (client-compatible), port released

Caught during verification: returning the apply report from service.delete would have broken Client::delete_service, which deserializes into () — deletes keep a null response and still reload.

Tests: 6 new in the supervisor (plan ordering, paused/disabled release, deterministic duplicate-port loser, port-release-after-await, propagation coverage). construct-daemon --lib 495, construct-cli --bins 1314, construct-protocol 1, workspace builds clean. The test asserting the old restart note failed loudly as intended and was updated.

Specs: adds 0173-service-definitions-apply-without-restart; amends 0163's "V1 does not promise runtime definition reload".

Touches crates/daemon, crates/protocol, crates/cli, web assets → all in the construct binary.

Definitions were read once at boot, so every edit — over IPC or by hand —
did nothing until the daemon restarted. Some of that was worse than a
missing feature: paused was consulted only at startup, so pausing a
running service left it serving, and rotating a credential left the old
one working. Both UIs asserted the restriction in prose rather than
reading restart_required, which nothing consumed.

A supervisor task now owns every listener. Definitions are read where they
are used, so routing, paused, and the approval timeout apply to the next
request and a rotated credential applies without touching the socket; a
port change rebinds, and pausing or detaching releases the port. Edits
made in the config directory are noticed too, since that is a documented
hand-editable surface.

Reload is all or nothing: a definition that fails to parse leaves the
running configuration untouched, which is also what makes watching the
directory safe against a file caught mid-write. Stops complete before
starts so two channels can exchange ports, and cancelling a listener stops
it accepting without interrupting a request that already has an agent turn
behind it.

Instruction, harness, model, cwd, and sandbox stay next-session. A harness
assembles its system prompt at startup and skips its seed prompt on
resume, so there is no honest way to re-instruct a live conversation; the
propagation classes say so rather than implying otherwise. They are
published as shared data so the promise beside a field is the one the
daemon keeps, and restart_required is replaced by a report of the socket
work actually performed.
The reload loop and the per-request config reads were only verified by
hand against a running daemon, so nothing would catch a regression.

Adds eight tests over the real reload path — files on disk, a real
session manager, real sockets — covering shared-state identity across a
reload, all-or-nothing parsing, services appearing and disappearing,
binding/moving/releasing a channel, and one service handing a port to
another in a single reload. Three more cover the mechanism the listener
never sees: a rotated credential, pausing or disabling a channel, and an
edit preserving routed sessions and delivery history.

Each was checked against a deliberately broken build so it fails for the
reason it claims. That found two problems here:

The stop-before-start test was vacuous. Reversing the plan's order did
not fail it, because a port swap produces two rebinds and no start to
order against. The ordering is in fact enforced by the executor's two
passes rather than by the plan, so the comment claiming otherwise is
corrected and the test now uses a handover, where a stop and a start
genuinely coexist.

The port-claiming tests raced each other for the same ephemeral port —
free_port reports one that was free a moment ago — and failed roughly one
run in six. They now serialize.
@edwin-zvs

Copy link
Copy Markdown
Contributor Author

Added regression coverage

The reload loop and the per-request config reads were only verified by hand against a running daemon, so nothing would have caught a regression. Service tests go from 6 to 14.

Over the real reload path (files on disk, real SessionManager, real sockets):

  • a reload reuses a known service's state — the invariant whose violation strands live conversations and admits duplicate deliveries
  • a broken definition changes nothing
  • services appear and disappear with their definitions
  • a channel binds, moves, and releases across reloads
  • one service hands a port to another in a single reload

The mechanism the listener never sees:

  • a rotated credential takes effect without rebinding (and a detached channel authenticates nobody)
  • pausing or disabling stops a channel serving
  • an edit preserves routed sessions and delivery history

Each was checked against a deliberately broken build, so it fails for the reason it claims: rebuilding shared state on reload, skipping the await before rebind, removing the release pass, freezing the token snapshot, ignoring paused, and clearing dedup on edit. All six mutations failed the intended test.

That exercise found two problems in this PR:

  1. The stop-before-start test was vacuous. Reversing the plan's ordering did not fail it — a port swap produces two rebinds and no Start to order against. Investigating showed the guarantee is enforced by the executor's two passes, not by plan, so the comment crediting the plan is corrected and the test now uses a port handover, where a stop and a start genuinely coexist. There is also an end-to-end version that fails with EADDRINUSE if the release pass is removed.
  2. A flaky test. The suite failed once at 502/1 while passing when filtered — the port-claiming tests racing for the same ephemeral port, since free_port reports one that was free a moment ago. They now serialize; 5/5 clean afterward.

construct-daemon --lib 503, construct-cli --bins 1314, workspace builds clean.

@edwin-zvs
edwin-zvs merged commit a3f07c7 into main Aug 1, 2026
1 of 2 checks passed
@edwin-zvs
edwin-zvs deleted the service-live-reload branch August 1, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant