Repository navigation
fix(gateway): enforce a shared memory-preparation deadline with safe fallback #1741
Description
Parent incident: #1708
Priority: Integration after the individual cancellation/read contracts
Confirmed code gap
On the normal conversation path, the 300-second foreground abort scope is created after LTM selection, distillation prewarm and gradient processing. Nested optional operations can each consume their own timeout or ignore cancellation. The 8 September trace spent 315.4 s before upstream.
Adjacent timeout/error lines have no request correlation, so this issue must not assume that a later timeout belongs to that same turn.
Baseline: normal conversation path and foreground scope.
Implementation plan
- Establish one absolute request lifetime at admission and a separately configurable, short post-decode memory-preparation budget. Keep ingress limits/timeouts and upstream-generation policy explicit rather than giving every nested operation a fresh full timeout.
- Propagate remaining time/cancellation through stable/context LTM, query embeddings, read-worker prewarm and prompt preparation. Reuse fix(ltm): bound and cancel foreground query embedding waits #1735, fix(reads): keep heavy worker failures off the gateway thread #1736 and perf(workers): bound read-pool queues and prioritize foreground requests #1739. Through fix(ltm): bound and cancel foreground query embedding waits #1735, use fix(embeddings): decouple request cancellation from worker lifetime #1745's independent embedding-worker lifetime; a preparation deadline expires the job/caller, not the worker's health.
- On optional enrichment exhaustion, assemble from a validated accepted source window/pinned knowledge plus the current suffix; optionally use bounded FTS only when time remains. Preserve current instructions, pending tool pairs, provenance, ownership and token limits.
- If no safe window can be assembled, return a sanitized protocol-correct preparation error. Never silently send an oversized or semantically incomplete request, or advance a checkpoint before the response/temporal transaction commits.
- Check elapsed time at synchronous phase boundaries. A timer cannot interrupt synchronous JS/native work: avoid starting unbounded work under the fallback contract and retain profiling evidence for any remaining cold-path blocker.
- Ensure shared/single-flight work is not globally canceled merely because one waiter disconnects; cancel its job interest when no legitimate waiter remains. Use fix(embeddings): decouple request cancellation from worker lifetime #1745's actual-execution accounting: final-waiter departure does not release native capacity or retire a healthy worker.
Acceptance criteria
- Time consumed by nested optional operations is bounded by one preparation budget, not the sum of individual timeouts.
- Slow/unavailable embeddings or read workers take a safe bounded path and do not continue admitting abandoned work.
- Warm/cold/resume/rewind fixtures preserve exact current-turn/provenance/tool and durable-write guarantees.
- Client cancellation, budget expiry and upstream timeout remain distinguishable and cannot reset the absolute deadline.
- Mixed-load results from test(perf): add a reproducible long-session workload with background pressure #1733 measure against the stated p95 <2 s post-decode overhead target; this is a reference-machine goal, not a hardware-independent guarantee.
Scope boundary
One pipeline-integration PR after the prerequisite contracts. Cached gradient counts (#1737) improve the result but are not a hard dependency. Do not increase timeouts to hide synchronous work or redesign upstream model inference.
Hard prerequisites
- fix(ltm): bound and cancel foreground query embedding waits #1735 (which depends on fix(embeddings): decouple request cancellation from worker lifetime #1745)
- fix(reads): keep heavy worker failures off the gateway thread #1736
- perf(workers): bound read-pool queues and prioritize foreground requests #1739
Activity
fix(gateway): enforce a shared memory-preparation deadline (#1892)