Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ export function resolveConcurrencyLimit(
* which apply valid overrides verbatim), clamped to a hard ceiling.
*
* Three time bounds act on a dispatch and they are NOT redundant:
* - `stallMs` (60s default) — no *progress* for this long ⇒ abort + retry.
* - `stallMs` (3 min default) — no *progress* for this long ⇒ abort + retry.
* Held while a tool is in flight, so a slow tool is not a stall.
* - `max_time_minutes` (this) — total wall time for ONE attempt, stalled or
* not. Bounds the case the watchdog cannot see (a model that keeps
Expand Down
2 changes: 1 addition & 1 deletion packages/core/src/agents/runtime/workflow-sandbox.ts
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,7 @@ export interface WorkflowAgentOpts {
/**
* P-stall: per-call stall-watchdog timeout in milliseconds. The dispatch
* is aborted + retried (up to 3 attempts) after this many ms of no
* subagent progress (with no tool in flight). Defaults to 60_000 (env
* subagent progress (with no tool in flight). Defaults to 180_000 (env
* override `QWEN_CODE_WORKFLOW_STALL_SECONDS`). `0` disables the watchdog
* for this call.
*/
Expand Down
96 changes: 85 additions & 11 deletions packages/core/src/agents/runtime/workflow-stall.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ import {
MAX_STALL_ATTEMPTS,
MAX_WORKFLOW_STALL_MS_ENV,
} from './workflow-stall.js';
import { DEFAULT_RETRY_OPTIONS } from '../../utils/retry.js';
import { getRetryDelayMs } from '../../utils/retryPolicy.js';

describe('resolveStallMs', () => {
it('uses the per-call override when positive', () => {
Expand Down Expand Up @@ -46,6 +48,51 @@ describe('resolveStallMs', () => {
it('ignores a negative per-call value (falls through to default)', () => {
expect(resolveStallMs(-5, {})).toBe(DEFAULT_STALL_MS);
});

// The default is not an arbitrary round number: it has to outlast the
// transport's own silent retry ladder, or a request that is retrying exactly
// as designed reads as a stall. `DEFAULT_RETRY_OPTIONS` in utils/retry.ts
// sleeps 1.5s, 3s, 6s, 12s, 24s, 30s between attempts, and agent-core
// consumes each `retry` stream event without emitting anything the watchdog
// counts as progress — so the whole ladder is one silent stretch.
//
// Asserting the relationship rather than the literal keeps this meaningful if
// either number is retuned later.
//
// The ladder is DERIVED from `DEFAULT_RETRY_OPTIONS`, not hand-copied: a
// local literal would keep this test green while a retune of the real
// options pushed the real ladder past the window — the exact false-stall
// regression this test exists to prevent. Mirrors retryWithBackoff's error
// path: `maxAttempts - 1` sleeps, `currentDelay` doubling from
// `initialDelayMs` under the `maxDelayMs` cap, each sleep run through
// `getRetryDelayMs` with the ±30% jitter that path applies.
const transportLadderMs = (random: () => number) => {
const { maxAttempts, initialDelayMs, maxDelayMs } = DEFAULT_RETRY_OPTIONS;
let currentDelay = initialDelayMs;
let total = 0;
for (let sleep = 1; sleep < maxAttempts; sleep++) {
total += getRetryDelayMs({
attempt: 1,
initialDelayMs: currentDelay,
maxDelayMs,
jitterRatio: 0.3,
random,
});
currentDelay = Math.min(maxDelayMs, currentDelay * 2);
}
return total;
};

it('outlasts the transport retry ladder it has to survive', () => {
const nominal = transportLadderMs(() => 0.5); // jitter cancels out
const worstCase = transportLadderMs(() => 1); // every sleep +30%, then capped
expect(nominal).toBe(76_500);
expect(worstCase).toBe(89_250);
// The window has to outlast the ladder on an UNLUCKY run, not just the
// nominal sum: a `DEFAULT_STALL_MS` retuned into the (76.5s, 89.25s] band
// would false-trip under jitter while a nominal-only assertion stayed green.
expect(DEFAULT_STALL_MS).toBeGreaterThan(worstCase);
});
});

describe('attachStallWatchdog', () => {
Expand All @@ -56,15 +103,19 @@ describe('attachStallWatchdog', () => {
vi.useRealTimers();
});

it('fires after stallMs of no activity once the first response has arrived', () => {
it('fires after stallMs of silence once armed (ROUND_START)', () => {
const emitter = new AgentEventEmitter();
const controller = new AbortController();
const wd = attachStallWatchdog(emitter, controller, 1000);
// #8: not armed until the first progress event (the time-to-first-response
// window is not a stall) — so advancing past stallMs here does nothing.
// Not armed until the first progress event — so advancing past stallMs
// here does nothing. In a real dispatch that event is ROUND_START, which
Comment thread
qqqys marked this conversation as resolved.
// fires before the request reaches the wire (see the doc comment on
// `attachStallWatchdog`), so this pre-arm silence is only round 1's
// pre-generator work, NOT the time-to-first-token window — that window is
// watched, and is pinned by the test below.
vi.advanceTimersByTime(2000);
expect(wd.stalled()).toBe(false);
// First response arrives → watchdog arms; then silence trips it.
// ROUND_START arrives → watchdog arms; then silence trips it.
emitter.emit(AgentEventType.ROUND_START, {} as never);
vi.advanceTimersByTime(999);
expect(wd.stalled()).toBe(false);
Expand All @@ -75,18 +126,39 @@ describe('attachStallWatchdog', () => {
wd.dispose();
});

it('does NOT fire during the time-to-first-response window (#8)', () => {
// Replaces a test that asserted the watchdog does not fire during the
// time-to-first-response window. That test never emitted ROUND_START, so it
Comment thread
qqqys marked this conversation as resolved.
Comment thread
qqqys marked this conversation as resolved.
// only proved that a silent emitter does not trip a watchdog that was never
// armed — and it encoded a model of the transport that is not true. In a real
// dispatch ROUND_START has already fired by this point: `sendMessageStream`
// returns a lazily iterated generator, so the `await` on it resolves before
// the request reaches the wire, and agent-core emits ROUND_START on the very
// next line. These two tests pin what actually happens.
it('does not arm before the first progress event', () => {
const emitter = new AgentEventEmitter();
const controller = new AbortController();
const wd = attachStallWatchdog(emitter, controller, 1000);
// A reasoning model thinking for a long time before the first token emits
// no events; that must not be treated as a stall.
// Nothing emitted: the timer was never armed, so nothing can elapse.
vi.advanceTimersByTime(10_000);
expect(wd.stalled()).toBe(false);
expect(controller.signal.aborted).toBe(false);
wd.dispose();
});

it('DOES count the time-to-first-token window, because ROUND_START precedes the request', () => {
const emitter = new AgentEventEmitter();
const controller = new AbortController();
const wd = attachStallWatchdog(emitter, controller, 1000);
// Exactly what agent-core does: emit ROUND_START immediately after
// `await sendMessageStream(...)` resolves — i.e. before any bytes are sent.
emitter.emit(AgentEventType.ROUND_START, {} as never);
// The provider is still connecting/queueing/thinking; no deltas yet.
vi.advanceTimersByTime(1001);
expect(wd.stalled()).toBe(true);
expect(controller.signal.reason).toBe('stalled');
wd.dispose();
});

it('resets the timer on a progress event', () => {
const emitter = new AgentEventEmitter();
const controller = new AbortController();
Expand Down Expand Up @@ -157,8 +229,9 @@ describe('runStallResilient', () => {
emitter: AgentEventEmitter,
): Promise<string> => {
calls += 1;
// Emit a first response event so the watchdog arms (#8: the time-to-
// first-response window is not a stall), then go silent → it trips.
// Emit ROUND_START so the watchdog arms — in a real dispatch that fires
// before the request reaches the wire, so the time-to-first-token window
// IS watched — then go silent → it trips.
emitter.emit(AgentEventType.ROUND_START, {} as never);
await new Promise<void>((resolve) => {
if (signal.aborted) return resolve();
Expand Down Expand Up @@ -187,8 +260,9 @@ describe('runStallResilient', () => {
): Promise<string> => {
calls += 1;
if (calls < 2) {
// First attempt stalls: emit a first response event to arm the
// watchdog (#8), then go silent until it aborts.
// First attempt stalls: emit ROUND_START to arm the watchdog — in a
// real dispatch that fires before the request reaches the wire — then
// go silent until it aborts.
emitter.emit(AgentEventType.ROUND_START, {} as never);
await new Promise<void>((resolve) => {
if (signal.aborted) return resolve();
Expand Down
64 changes: 49 additions & 15 deletions packages/core/src/agents/runtime/workflow-stall.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
* @fileoverview Stall watchdog + retry for workflow agent dispatches. A
* workflow `agent()` can hang indefinitely if the model loops, the provider
* stalls mid-stream, or a tool never returns. The subagent's own
* `max_time_minutes` (10 min) is a coarse backstop; the stall watchdog is
* finer-grained: it aborts a dispatch after `stallMs` (default 60s) of NO
* `max_time_minutes` (10 min, per attempt) is a coarse backstop; the stall
* watchdog is finer-grained: it aborts a dispatch after `stallMs` (default 3 min) of NO
* observable progress, and the resilient wrapper retries up to
* `MAX_STALL_ATTEMPTS` times before abandoning.
*
Expand Down Expand Up @@ -40,8 +40,31 @@ import { AgentEventEmitter, AgentEventType } from './agent-events.js';
import { createDebugLogger } from '../../utils/debugLogger.js';
import { parsePositiveIntegerEnv } from '../../utils/env.js';

/** Default stall timeout: 60s of no progress (with no tool in flight). */
export const DEFAULT_STALL_MS = 60_000;
/**
* Default stall timeout: no progress (with no tool in flight) for this long
* ends the attempt.
*
* Sized against the `retryWithBackoff` silent retry ladder rather than against
* a guess at model latency. That ladder is the binding case, not the only
* watchdog-invisible wait: stream-side rate-limit sleeps
* (`RATE_LIMIT_RETRY_OPTIONS` in geminiChat.ts — 60s/120s/240s/300s, so two
* consecutive sleeps already reach 180s), a provider `Retry-After` honored
* unclamped on the normal HTTP path, and unattended-mode persistent backoff
* (up to 5 min per exponential sleep — but a provider `Retry-After` on that
* path is capped only at `PERSISTENT_CAP_MS`/6h, not at the 5 min
* `PERSISTENT_MAX_BACKOFF_MS`, so one 429 carrying `Retry-After: 7200` sleeps
* two hours) can all exceed this window and read as a stall on a
* request that is retrying exactly as designed. Making transport retries
* visible to the watchdog is the follow-up; this window does not cover them.
* `retryWithBackoff` sleeps 1.5s, 3s, 6s, 12s, 24s then
* 30s between attempts (`DEFAULT_RETRY_OPTIONS` in utils/retry.ts), and
* `agent-core` consumes each `retry` stream event without emitting anything the
* watchdog counts as progress. A plain 429/5xx ladder is therefore 76.5s of
* watchdog-invisible silence on a request that is healthy and retrying exactly
* as designed — comfortably past the previous 60s value, which elapsed during
* the ladder's sixth sleep.
*/
export const DEFAULT_STALL_MS = 180_000;

/** Total attempts (initial + retries) for a single `agent()` dispatch. */
export const MAX_STALL_ATTEMPTS = 3;
Expand Down Expand Up @@ -89,13 +112,22 @@ export interface StallWatchdogHandle {
* flight the timer is held (a long tool call is not a stall). When the
* timer elapses with no in-flight tool, it fires `controller.abort('stalled')`.
*
* The watchdog arms on the FIRST progress event, not at attach time. The
* time-to-first-response window — connection setup, server-side queueing, and
* a reasoning model's pre-first-token thinking — emits no events (`ROUND_START`
* fires only AFTER `await sendMessageStream` resolves), so counting it would
* false-trip on a healthy-but-slow first response and waste 3× tokens on the
* retry loop. That window is instead bounded by the subagent's own
* `max_time_minutes`; the watchdog's job is post-first-response streaming stalls.
* The watchdog arms on the first progress event rather than at attach time, but
* that excludes far less than it appears to. `ROUND_START` is emitted as soon as
* `await sendMessageStream(...)` resolves — and that call returns a lazily
* iterated async generator, so it resolves BEFORE the request reaches the wire;
* the generator body issues it on first iteration. The deferred arm therefore
* only skips round 1's pre-generator work (the send-lock drain, route
* resolution, and any auto-compaction), not the request itself. Connection
* setup, server-side queueing, and a reasoning model's pre-first-token thinking
* all elapse with the timer already running.
*
* So `stallMs` must be wide enough to cover a healthy first response, not just a
* mid-stream gap. The binding case this window is sized against is the
* `retryWithBackoff` silent retry ladder — see `DEFAULT_STALL_MS`, which also
* names the longer waits (stream-side rate-limit sleeps, an unclamped
* `Retry-After`, unattended backoff) that remain invisible to it. Once the provider streams anything at all, including
* thought deltas, `STREAM_TEXT` resets the timer.
*
* A `stallMs` of 0 means "no watchdog" — this returns an inert handle.
*/
Expand Down Expand Up @@ -161,10 +193,12 @@ export function attachStallWatchdog(
emitter.on(AgentEventType.TOOL_CALL, onToolCall);
emitter.on(AgentEventType.TOOL_RESULT, onToolResult);

// Intentionally NOT armed here. The first `onActivity` (the first response
// event of round 1) arms it, so time-to-first-response is not counted as a
// stall (see the doc comment); first-response hangs are bounded by the
// subagent's `max_time_minutes`.
// Intentionally NOT armed here — the first `onActivity` arms it. Note this
// defers arming only past round 1's pre-request work, NOT past the request:
// `ROUND_START` fires before the call is on the wire (see the doc comment).
// A first-response hang is therefore caught by this watchdog, not left to
// the subagent's `max_time_minutes` (which is per attempt and resets on
// every stall retry, so it never bounds the sequence).

return {
stalled: () => fired,
Expand Down
8 changes: 3 additions & 5 deletions packages/core/src/tools/workflow/workflow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,11 +127,9 @@ const WORKFLOW_PARAM_SCHEMA = {
'`stallMs` (number, ms): a no-progress watchdog, not a wall-clock cap. ' +
'The dispatch is aborted and retried (up to ' +
`${MAX_STALL_ATTEMPTS} attempts total) after this many milliseconds ` +
'with no observable subagent progress once progress has begun ' +
'(a dispatch that produces no first response is bounded by the ' +
'subagent time cap, not this watchdog); the timer is suspended ' +
'while a tool is in flight, so a legitimately slow tool is not ' +
'a stall. ' +
'with no observable subagent progress — including before the first ' +
'response arrives; the timer is suspended while a tool is in flight, ' +
'so a legitimately slow tool is not a stall. ' +
`Default ${DEFAULT_STALL_MS} (override via \`${MAX_WORKFLOW_STALL_MS_ENV}\`, whole seconds); \`0\` disables the watchdog. Wall time ` +
'per attempt is bounded separately. ' +
'Workflow subagents always have SendMessage / Monitor / EnterPlanMode / ExitPlanMode ' +
Expand Down
7 changes: 6 additions & 1 deletion packages/core/src/utils/retry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,12 @@ export interface RetryOptions {
onRetry?: (info: RetryAttemptInfo) => void;
}

const DEFAULT_RETRY_OPTIONS: RetryOptions = {
/**
* Default ladder for the normal HTTP retry path. Exported so callers that
* must outlast it — the workflow stall watchdog sizes `DEFAULT_STALL_MS`
* against it — can derive the ladder rather than hand-copy it.
*/
export const DEFAULT_RETRY_OPTIONS: RetryOptions = {
maxAttempts: 7,
initialDelayMs: 1500,
maxDelayMs: 30000, // 30 seconds
Expand Down
Loading