fix(responses): give the bounded JSON read a first-byte deadline distinct from inter-chunk inactivity (#1065)

[lidge-jun]

Summary

• Fixes DeepSeek V4 Flash returns local 502 after 30s on bounded JSON path (post-#1026) #1065: DeepSeek V4 Flash requests intermittently died with a local 502 upstream JSON response stalled before completing after ~30.5s. Root cause: readBoundedResponseBody armed its 30s inactivity timer at reader creation — before the first body byte. A thinking model that flushes chunked headers promptly but computes >30s before emitting its first JSON byte was indistinguishable from a stalled upstream. Every deepseek-v4-flash turn takes this bounded JSON path because the registry deliberately pins modelResponsesUpstreamStreaming: false for it ([Bug] DeepSeek V4 Flash Responses route stalls after tool calls #875 terminal-event unreliability), so slow-thinking turns at high/max effort crossed the window and failed locally.
• BoundedBodyOptions gains firstByteTimeoutMs, defaulting to inactivityTimeoutMs so every existing caller is behavior-identical (web-search loop, images loop, upstream-http-error, kiro-retry, auth-api, quota-rejection, and the other two core call sites). Only the bounded JSON upstream read opts in, passing the existing 180s total budget as the first-byte allowance — the total wall-clock deadline still bounds a genuinely dead upstream, and the 30s inter-chunk stall protection is unchanged once bytes flow.
• The options are exported as a named policy const so a regression test pins the call-site wiring, and the timeout comment now explains the early-header/late-first-byte case instead of implying pre-header generation was the only wait.
• Resource tradeoff (recorded in the devlog unit): a header-only stall can now hold a turn slot for up to 180s instead of 30s; buffer memory stays bounded (64 KiB initial buffer × 256 active-turn cap ≈ 16 MiB worst case).

Verification

• bun run typecheck — clean.
• bun test tests/bounded-body.test.ts — 21 pass, 0 fail (new: first byte delayed past inactivity but under first-byte deadline completes; no first byte before the deadline still fails closed; on-time first byte with a later inter-chunk stall still times out; defaulted option behaves identically to today; call-site policy pin asserting firstByteTimeoutMs === totalTimeoutMs for the bounded JSON read).
• Red ablation recorded: the delayed-first-byte test fails without the fix (Expected: "first byte" / Received: "" — 0 pass, 1 fail).
• Smoke: bun test tests/responses-shadow-intercept.test.ts — 14 pass, 0 fail.
• bun run privacy:scan — passed.

Closes #1065.

Checklist

• Scope stays focused and avoids unrelated cleanup.
• Docs or release notes were updated when needed. (Behavior is internal reliability; the devlog unit carries the RCA and design.)
• Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. (Fail-closed semantics preserved; all bounds still enforced.)

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of slow upstream responses by applying a dedicated timeout for receiving the first byte.
  • Continued enforcing inactivity limits between subsequent response chunks.
  • Standardized timeout behavior for non-streaming JSON responses, reducing delays when upstream services do not respond promptly.

[github-actions]

✅ Deterministic PR hygiene checks passed.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: cb794e44-4075-450b-9260-62c223b95661

📥 Commits

Reviewing files that changed from the base of the PR and between ad38b5f and c96abb7.

📒 Files selected for processing (3)

• src/lib/bounded-body.ts
• src/server/responses/core.ts
• tests/bounded-body.test.ts

---

📝 Walkthrough

Walkthrough

Changes

Bounded JSON response timeout

Layer / File(s)	Summary
Bounded body timeout behavior src/lib/bounded-body.ts, tests/bounded-body.test.ts	BoundedBodyOptions adds firstByteTimeoutMs. Initial reads use this timeout, while later chunks use inactivityTimeoutMs. Tests cover delayed first bytes, timeout fallback, and later-chunk inactivity.
Upstream JSON timeout wiring src/server/responses/core.ts, tests/bounded-body.test.ts	UPSTREAM_JSON_BODY_READ_OPTIONS centralizes JSON body limits and sets firstByteTimeoutMs to the total timeout. The JSON response reader uses this shared configuration.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant UpstreamJSONResponse
  participant ResponsesCore
  participant BoundedBodyReader
  ResponsesCore->>BoundedBodyReader: readBoundedResponseBody(UPSTREAM_JSON_BODY_READ_OPTIONS)
  BoundedBodyReader->>UpstreamJSONResponse: wait up to firstByteTimeoutMs
  UpstreamJSONResponse-->>BoundedBodyReader: first JSON body chunk
  BoundedBodyReader->>UpstreamJSONResponse: enforce inactivityTimeoutMs for later chunks
  UpstreamJSONResponse-->>BoundedBodyReader: remaining JSON body chunks
  BoundedBodyReader-->>ResponsesCore: bounded JSON response body

Loading

Suggested reviewers: ingwannu, wibias, luvs01

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: a distinct first-byte deadline for bounded JSON reads.
Linked Issues check	✅ Passed	The changes satisfy [#1065] by extending the first-byte deadline while preserving inter-chunk inactivity and total response limits.
Out of Scope Changes check	✅ Passed	All changes support [#1065], including timeout configuration, call-site wiring, and regression tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/1065-bounded-body-first-byte

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}