fix(embeddings): decouple request cancellation from worker lifetime #1745

Parent incident: #1708

Priority: First implementation task, selected by the owner. Split the worker-lifetime contract out of #1735 so it can ship before short foreground deadlines are connected.

Problem

In the current local embedding pool, EmbeddingRequestAbortedError causes EmbeddingPool.embed() to call retireSlot(). That shuts down the assigned worker, rejects its other pending requests and discards its loaded model. One request expiring while queued can therefore disrupt unrelated work on a healthy worker.

The worker protocol currently has embed/shutdown messages and result/error replies, without per-job cancellation or execution-start acknowledgement. LocalProvider also waits for shared initialization before attaching the request abort listener.

The 20:00 UTC temporal drain reached its 60 s deadline at stage=embed and settled after 61.923 s. That supports prioritizing cancellation isolation; it does not prove worker retirement caused the earlier 315 s foreground stall.

Verified source at aefb5612f60f: provider and pool, worker, protocol, temporal scheduler.

Required behavior

Runtime shutdown and explicit model/config replacement remain legitimate lifecycle operations.

Implementation plan

1. Separate caller settlement from execution ownership. Track queued/dispatched/running/terminal work and worker generation independently of whether its caller still awaits a result. Ordinary caller abort must not mark a worker unhealthy or consume its initialization-failure retry budget.
2. Prefer retaining waiting jobs on the host and dispatching at most one native batch per worker, preserving query priority. An alternative must provide acknowledged per-job cancellation before execution. A cancel message alone is insufficient when the worker is blocked or drains queued batches before processing that message.
3. Make shared initialization waits cancellable per caller. Add the minimal execution-start/completion/cancel acknowledgement needed for race-safe ownership. Bound retained cancelled-job state and remove listeners/input buffers when safe.
4. Give the runtime independent finite initialization and execution watchdogs. A short request deadline detaches that caller; it does not become the worker watchdog. Watchdogs must distinguish readiness, dispatch and actual service, must not reset when new callers arrive, and must not mistake queue age for stuck inference.
5. Preserve actual-work accounting through completion or confirmed worker exit, including retirement and memory-gated replacement. Use bounded shutdown/recovery for a real failure or stuck execution; preserve existing OOM/WASM recovery and healthy siblings.
6. Preserve durable temporal/document jobs and generation guards. Cancelling their waiter must not admit a concurrent retry of the same still-running job. Late results cannot commit cancelled work or write through a replaced database/runtime. Drain/close waits for execution quiescence, not merely settled caller promises.

Acceptance criteria

• Queue B behind blocked job A on one warmed worker, cancel B, then release A: B never reaches inference, A succeeds, no restart/model reload occurs, and another request reuses that worker.
• Cancelling one of two initialization waiters settles that waiter promptly while the other completes; already-aborted input starts no work.
• Cancellation during executing inference settles the caller promptly without releasing execution capacity. Completion frees it once; late/duplicate replies and worker generations cannot settle or release twice.
• A stuck initialization or executing batch triggers its independent watchdog and bounded recovery even after all callers detach. Unrelated healthy workers remain usable.
• Worker retirement/replacement remains within the memory/capacity budget; ordinary cancellation does not cause replacement churn.
• A temporal deadline followed by retry does not duplicate still-running work, drop durable jobs or commit late cancelled results. Explicit shutdown/config replacement drains correctly.
• Regression tests cover blocked-worker message delivery, cancellation/start races, repeated cancellation and completion/exit races. Update the existing pool test that currently expects an ordinary abort to retire its worker.

Scope and follow-ups

One local provider/pool/protocol lifecycle PR using the existing worker threads and durable queue. No child-process migration, model change, reserved query worker, global scheduler, or new queue backend.

#1735 then wires foreground LTM signals, short deadlines and FTS fallback to this contract. #1732 owns richer timing/metrics; this issue includes only the control acknowledgements and fixed failure reasons needed for correctness. #1740 owns background admission, #1734 investigates actual provider stalls, and #1672 supplies broader runtime lifecycle conventions.

Hard prerequisites

None. Implement and reproduce with deterministic worker fixtures; live profiling and the broader lifecycle refactor are not prerequisites.