[Bug]: Streaming assistant deltas rescan full thread activity history #4008

Activity

1. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   🚧 In Progress

   on Jul 22, 2026
2. VincentShipsIt commented on Jul 30, 2026

   

   VincentShipsIt

   on Jul 30, 2026

   More actions

   Desktop repro data (macOS, packaged app) — matches this issue closely

   Environment

   • T3 Code (Alpha) 0.0.31 (com.t3tools.t3code)
   • macOS desktop, local backend on 127.0.0.1:3773
   • Grok Build (ACP) + Codex + Claude Agent all configured; slowdown reproduced during multi-tool Grok turns
   • enableAssistantStreaming: true in ~/.t3/userdata/settings.json at capture time

   Local profile at capture (~/.t3/userdata/state.sqlite)

   Metric	Value
   DB file size	587 MB
   orchestration_events	151,749
   thread.message-sent	127,380 (~84%) / ~38 MB payload
   thread.activity-appended	22,813 / ~174 MB payload
   Projected activities	22,813 / ~168 MB
   Projected messages	2,286 / ~0.5 MB

   Heaviest threads (pre-trim)

   Title	Activities	Events	Status
   tests coverage	14,709	102,002	archived+deleted (still on disk)
   QA	2,461 (~80 MB activities)	11,159	active
   /deploy	2,246	16,462	archived+deleted
   Fix Messages Link Routing	1,062	4,713	active
   QA (other)	981 (~20 MB)	5,688	active

   User-visible symptom
   Grok Build turns start fine, then after a couple of tool-call rounds in the same thread the UI becomes dramatically slower (text trickle / long gaps between tool steps). Fresh threads feel normal; long agentic threads degrade mid-turn. This is not “Grok CLI is slow” — terminal Grok is fine; T3 falls over as history grows.

   Why this looks like #4008

   • Streaming was on → event log dominated by thread.message-sent deltas (one event per stream chunk).
   • Tool rounds add large thread.activity-appended payloads (activity payload mass ≫ message text mass).
   • Slowdown only appears after the thread accumulates activities, consistent with per-delta work that re-touches full activity history / shell summary (as described here), not with provider token latency.

   Server traces (supporting)

   • Large volumes of runProjectorForEvent + sql.execute during agent turns
   • Long-lived ws.rpc.orchestration.subscribeThread spans (minutes–hours class in duration fields)
   • Separate Grok ACP noise also present historically (Cannot convert skills-reload to a BigInt / model discovery timeout) — orthogonal, not the mid-turn regression

   Workaround that recovered the machine

   1. Full quit of the app
   2. SQLite .backup, then purge soft-deleted thread rows + wipe history on fat active threads (kept thread shells + provider session rows)
   3. Set enableAssistantStreaming: false
   4. VACUUM → DB 587 MB → 288 KB

   Happy to re-capture timing of assistant-delta projection (median/p95) on a synthetic thread if useful for a regression test. Cross-links: also matches client quadratic replay in #4596 and unbounded history hydration in #2761 / #3510.
3. DTavaszi commented on Aug 4, 2026

   

   DTavaszi

   on Aug 4, 2026

   More actions

   Confirmed a broader variant of this issue with assistant streaming disabled.

   A long-lived thread containing tens of thousands of activities made each thread.activity-appended dispatch spend roughly two seconds in refreshThreadShellSummary. ProviderRuntimeIngestion uses a single serial DrainableWorker for all provider events, so these slow events created cross-thread head-of-line blocking: an unrelated thread completed normally at the provider, but its response was not persisted and rendered by T3 Code for about seven minutes.

   This suggests the impact is not limited to assistant-delta streaming within the large thread. Routine activity events from one large thread can starve unrelated conversations globally.

   The completion was not lost or corrupted; it became visible after the ingestion backlog drained. In addition to making shell-summary projection incremental, partitioning ingestion by thread or moving expensive projections off the global serial path may prevent the cross-thread failure mode.
4. ognjeeen commented on Aug 6, 2026

   

   ognjeeen

   on Aug 6, 2026

   More actions

   Still reproducible on T3 Code 0.0.31 across two independent Windows installations.

   Additional measurements and controlled cleanup A/B data are documented in #5110:

   • one 10,545-character response produced 3,024 durable thread.message-sent events (median delta: 3 characters);
   • after cleanup, one database regrew from 282 KB to 218 MB and 93,943 events in approximately 25 hours;
   • removing only T3 Code's internal history immediately restored responsive streaming;
   • PRAGMA quick_check remained ok;
   • built-in thread deletion did not purge historical events, receipts, messages, or activities.

   This supports #4008 while suggesting that the shell-summary rescan is one amplifier within a broader per-delta persistence, queueing, and retention problem.
5. t3dotgg

   marked [Bug]: Assistant streaming causes full-thread projection scans for every text delta #5719 as a duplicate of this issue on Aug 22, 2026
6. t3dotgg

   closed this as completedin #8150on Aug 25, 2026