OpenAI-compatible stream idle timer ignores SSE keepalive chunks → false 'no data' timeouts + prefill retry livelock

[jojig-dao]

OpenAI-compatible stream idle timer ignores SSE keepalive chunks → false "no data" timeouts + prefill retry livelock

Environment

• jcode v0.76.0 (bbef1f6), macOS aarch64
• Provider: openai-compatible → local oMLX server (OpenAI-compatible chat completions, SSE streaming)
• Local model: 27B hybrid (prefill ~110–160 tok/s on an M4 Max), so large prompts legitimately spend minutes in prefill before the first content token

Summary

When streaming from a server with a long prefill, jcode aborts the stream with OpenRouter SSE stream timed out (no data for 600s) even though the server emits SSE keepalive chunks every ~9 seconds throughout the prefill. The idle timer appears to reset only on content-bearing deltas; keepalive chunks ("model":"keepalive", "created":0, empty delta.content) do not count as activity.

Worse, the "no data" deadline appears to be inherited across retry attempts rather than reset when a new stream opens.

Evidence (jcode log, 2026-08-15 local time)

• stream_idle_timeout_secs = 600 in config (confirmed live).
• Attempt 1: stream opened 21:51:31, timed out 21:55:16 (only 225 s later) reporting no data for 600s — 600 s before was 21:45:16, the open time of a previous request's stream.
• Attempt 2: opened 21:55:17, timed out 22:01:31 — exactly 600 s counted from attempt 1's open time.
• Same pattern for attempts 3–8 (22:05:17, 22:15:20, …): each deadline = previous attempt's start + 600 s.
• Server-side (oMLX) logs confirm keepalives were flowing and the prefill was progressing normally during every one of these "no data" windows.

Consequence: retry livelock

The server cannot resume an aborted prefill, so every client-side abort restarts the full prefill from token 0. With the inherited deadline, each attempt dies before prefill completes, and all 8/8 attempts fail. Real measurements from one session:

• 14,903-token prompt (~2–3 min prefill): answered only after 68 minutes and 8 attempts — and one attempt had already generated 33 chars of reply when the client timer rolled it back (Mid-stream retry rollback (attempt 7/8): discarding partial output (33 text chars, 0 tool calls)).
• 82,089-token prompt (~12–15 min prefill): never completed; 8/8 attempts timed out.

A related second issue: after switching the session's route away from the local model mid-turn, the old route's retry loop kept running in the background for ~1 hour (attempts at 21:55, 22:05, 22:15, 22:25, 22:35… after the route switch at 21:49), hammering the local server with 82k-token prefills. A turn cancel / route switch should cancel the retry loop.

Repro

1. Point openai-compatible at any server whose prefill exceeds the idle timeout (a stub that emits SSE keepalives then sleeps N minutes before the first content token works).
2. Send a prompt. Watch attempts die at the inherited deadline despite keepalives arriving every few seconds.

Suggested fixes

1. Count any received SSE frame — including model:"keepalive" chunks, comment frames, and empty deltas — as stream activity for idle-timeout purposes.
2. Reset the idle/no-data deadline per attempt when the new stream opens.
3. Cancel a turn's provider retry loop when the turn is cancelled or the session route is switched.

Happy to provide full jcode + server logs privately if useful.