Skip to content

fix(gateway): enforce a shared memory-preparation deadline with safe fallback #1741

Open

Description

@BYK

Parent incident: #1708

Priority: Integration after the individual cancellation/read contracts

Confirmed code gap

On the normal conversation path, the 300-second foreground abort scope is created after LTM selection, distillation prewarm and gradient processing. Nested optional operations can each consume their own timeout or ignore cancellation. The 8 September trace spent 315.4 s before upstream.

Adjacent timeout/error lines have no request correlation, so this issue must not assume that a later timeout belongs to that same turn.

Baseline: normal conversation path and foreground scope.

Implementation plan

  1. Establish one absolute request lifetime at admission and a separately configurable, short post-decode memory-preparation budget. Keep ingress limits/timeouts and upstream-generation policy explicit rather than giving every nested operation a fresh full timeout.
  2. Propagate remaining time/cancellation through stable/context LTM, query embeddings, read-worker prewarm and prompt preparation. Reuse fix(ltm): bound and cancel foreground query embedding waits #1735, fix(reads): keep heavy worker failures off the gateway thread #1736 and perf(workers): bound read-pool queues and prioritize foreground requests #1739. Through fix(ltm): bound and cancel foreground query embedding waits #1735, use fix(embeddings): decouple request cancellation from worker lifetime #1745's independent embedding-worker lifetime; a preparation deadline expires the job/caller, not the worker's health.
  3. On optional enrichment exhaustion, assemble from a validated accepted source window/pinned knowledge plus the current suffix; optionally use bounded FTS only when time remains. Preserve current instructions, pending tool pairs, provenance, ownership and token limits.
  4. If no safe window can be assembled, return a sanitized protocol-correct preparation error. Never silently send an oversized or semantically incomplete request, or advance a checkpoint before the response/temporal transaction commits.
  5. Check elapsed time at synchronous phase boundaries. A timer cannot interrupt synchronous JS/native work: avoid starting unbounded work under the fallback contract and retain profiling evidence for any remaining cold-path blocker.
  6. Ensure shared/single-flight work is not globally canceled merely because one waiter disconnects; cancel its job interest when no legitimate waiter remains. Use fix(embeddings): decouple request cancellation from worker lifetime #1745's actual-execution accounting: final-waiter departure does not release native capacity or retire a healthy worker.

Acceptance criteria

  • Time consumed by nested optional operations is bounded by one preparation budget, not the sum of individual timeouts.
  • Slow/unavailable embeddings or read workers take a safe bounded path and do not continue admitting abandoned work.
  • Warm/cold/resume/rewind fixtures preserve exact current-turn/provenance/tool and durable-write guarantees.
  • Client cancellation, budget expiry and upstream timeout remain distinguishable and cannot reset the absolute deadline.
  • Mixed-load results from test(perf): add a reproducible long-session workload with background pressure #1733 measure against the stated p95 <2 s post-decode overhead target; this is a reference-machine goal, not a hardware-independent guarantee.

Scope boundary

One pipeline-integration PR after the prerequisite contracts. Cached gradient counts (#1737) improve the result but are not a hard dependency. Do not increase timeouts to hide synchronous work or redesign upstream model inference.

Hard prerequisites

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions