Repository navigation
fix(agent-runtime): deduplicate persisted system events - #1499
ZxlDragonDoctor wants to merge 1 commit into
Conversation
Track normalized system-message identities across persistence and restoration so cloned events do not inflate the session journal. Keep events with distinct timestamps or tool declarations as separate history entries. Fixes vastsa#1477
muzimu217
left a comment
There was a problem hiding this comment.
Verified locally on macOS (worktree off current main): system-transcript journal tests 32/32 (4 files — including a test shaped exactly like the minimal repro in #1477), full agent-runtime suite 94 files / 1303 tests pass.
The fix implements the content-key direction from the issue discussion, plus the non-destructive recovery the issue asked for:
- One canonical key (
serializedSystemMessage: content text + tool declarations + timestamp) now feeds every dedup surface —persist,restore,remember,rememberCheckpoint, andisPersisted. The content-level set (serializedIds) is populated at persist/restore/remember, so the previously dead checkpoint path is alive again. - Semantics match the reporter's expectation precisely: identical payload with the same timestamp = the same durable event (the 256× duplicates collapse to one — covered by a clone-repro test), while the same payload at a new timestamp is a new event and is kept (dedicated test).
orderSystemRowsnow collapses exact duplicate rows during replay, so already-bloated v0.17.0 sessions recover non-destructively — rows stay on disk, context stops seeing the duplicates.
Two non-blocking observations: stored JSONL for old sessions stays bloated by design (a one-time compaction could reclaim disk, but that's optional scope beyond this fix), and rememberCheckpoint's old raw-JSON key form changed to the canonical one — no callers exist today (per the verification on #1477), so the semantic tightening is free.
fix scope, fits the current contribution window. Nothing blocking.
Problem
SystemTranscriptJournal deduplicates persisted messages by object identity, so cloned or reconstructed copies of the same system event can be appended again. Transcript replay also retained identical system rows, inflating model context.
Solution
Track a normalized serialized identity across persist, restore, remember, and checkpoint paths. Collapse exact duplicate system rows during replay while preserving messages whose timestamps or tool/model state differ.
Testing
node packages/agent-runtime/node_modules/vitest/vitest.mjs run packages/agent-runtime/src --config packages/agent-runtime/vitest.config.ts— 94 files passed; 1303 passed, 1 skipped.node 'C:\Users\朱小龙\AppData\Local\pnpm\.tools\pnpm\12.8.1\node_modules\pnpm\bin\pnpm.mjs' --filter @pi-desktop/agent-runtime typecheck— passed.link.exe; the available MinGW target still requires the MSVC host linker. Process-level persistence across a Host restart remains unverified.Issue alignment
Closes #1477. Repeated exact system events are collapsed; distinct timestamp or model-state events remain intact.