Skip to content

fix(semantic): persist filename-keyed memory summary cache#3364

Open
huangruiteng wants to merge 7 commits into
volcengine:mainfrom
huangruiteng:codex/fix-memory-summary-cache-1261
Open

fix(semantic): persist filename-keyed memory summary cache#3364
huangruiteng wants to merge 7 commits into
volcengine:mainfrom
huangruiteng:codex/fix-memory-summary-cache-1261

Conversation

@huangruiteng

Copy link
Copy Markdown
Contributor

Summary

  • persist per-file memory summaries in a deterministic .summary_cache.json keyed by the real filename
  • read that cache before falling back to best-effort .overview.md parsing, so unchanged files avoid repeated LLM summarization
  • write the cache together with .overview.md and .abstract.md under the same stale-message check and exact path-lock batch
  • cover cache reuse and atomic stale-write behavior with focused tests

This is a current-main replacement for #2636, whose patch no longer applies and whose test plan was not implemented.

Closes #1261.

Validation

  • pytest tests/storage/test_semantic_queue_memory_dedupe.py -q --no-cov — 8 passed
  • ruff format --check on the three changed files
  • ruff check on the three changed files
  • python -m compileall on the changed Python files
  • git diff --check

@huangruiteng

Copy link
Copy Markdown
Contributor Author

@qin-ctx 方便时请直接 review 这个既有 PR。它对应 #1261:把 memory summary cache 改为 filename-keyed durable cache,避免资源移动后重复触发 LLM summary;当前已与最新 main 保持可合并,GitHub checks 全绿,本地聚焦测试 8/8 通过,相关 Ruff 检查也通过。若你认为缓存归属或失效边界还需要收窄,请直接在 PR 里指出,我会继续在这个既有分支上收敛。

@huangruiteng

Copy link
Copy Markdown
Contributor Author

@qin-ctx I refined the existing #3364 branch in ddd0f0b after auditing the durable cache failure path. A transient per-file summary failure is represented as an empty summary; persisting that empty value would make later unchanged-file refreshes treat the failure as a permanent cache hit. The cache parser and serializer now reject empty or whitespace-only summaries, so the next incremental refresh retries the file while preserving valid filename-keyed entries. Validation: 9 focused semantic queue tests passed; Ruff check/format and diff hygiene passed. Please include this failure-retry boundary in the existing review when convenient.

@huangruiteng

huangruiteng commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@qin-ctx I tightened the existing sidecar migration boundary in 1b0e732. Once .summary_cache.json is structurally valid, it is now authoritative even when it has zero or partial usable entries; .overview.md is consulted only when the sidecar is missing, malformed, or has an unsupported version. This prevents an exact filename heading in an older overview from resurrecting a blank/failed cache entry and suppressing the intended retry, while preserving migration fallback for pre-sidecar directories. Validation: 11 focused tests passed, including authoritative blank-entry retry and invalid-version fallback; Ruff check/format, diff hygiene, and LoopX public-boundary scan passed. Please include this boundary in the existing review.

@huangruiteng

Copy link
Copy Markdown
Contributor Author

@qin-ctx #3364 最新 head 1b0e732c 的常规 PR checks 已通过;唯一未通过项是 API & CLI Integration run 29785422460。日志显示测试已到 93%,随后无密钥环境中的 Volcengine embedding/VLM 请求反复连接失败并分别等待约 182 秒,workflow 跑满 1 小时后被取消,没有出现代码断言失败。

我尝试直接 rerun,但 GitHub 返回需要仓库 admin 权限。烦请维护者重跑该 workflow;若重跑仍以相同外部 provider 超时结束,再把它作为 CI 环境稳定性问题处理,不先对本 PR 实现做无证据改动。

证据:https://github.com/volcengine/OpenViking/actions/runs/29785422460

@huangruiteng

Copy link
Copy Markdown
Contributor Author

@qin-ctx 状态更正:我已将 #3364 的现有分支无冲突合入最新 main 并推送 6336fc9,不再需要手动 rerun 先前被取消的 workflow;新的 API & CLI、dependency 与 plugin checks 均已自动进入 pending。当前 PR 相对 main 仍仅 3 个既定文件,本地 11 个聚焦测试、Ruff check/format 与 diff hygiene 全部通过。请以这个新 head 继续 review。

@huangruiteng

huangruiteng commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author
\n@qin-ctx 我在 #3364 既有分支继续做了 derived-sidecar 边界审计,并推送提交 519398c。此前新引入的 .summary_cache.json 虽由 semantic processor 生成,却不在 ContentWriteCoordinator 的 derived-file 禁止直写集合中;外部 create/replace 可写入合法 JSON,后续增量刷新会把它当权威 cache 复用。\n\n现在该文件名由 storage internal-names 统一定义,semantic processor 与 content-write 共用;直接 create/replace 均会按现有 .overview.md / .abstract.md 规则拒绝。验证:新增直写回归 2/2、既有 memory cache 回归 11/11、Ruff check/format 与 diff hygiene 均通过。请按最新 head 复核这个 cache 完整性边界。

@huangruiteng

Copy link
Copy Markdown
Contributor Author

@qin-ctx 补充推进到 42f26217,请有空帮忙 review 最新 head。

这轮在独立 worktree 对 current main 做合成 merge 审查时发现一个与本 PR sidecar 契约直接相关的问题:.summary_cache.json 在 snapshot restore 时会被当成普通 DETAIL 源文件重新入向量索引。现在已把它纳入 _NO_VECTOR_DERIVED,restore 的创建/删除路径都不会触发向量副作用,并补了对应分类回归。

验证:

  • tests/storage/test_semantic_queue_memory_dedupe.py:11 passed
  • tests/server/test_content_write_service.py -k memory_summary_cache:2 passed
  • 生产方法直接探针:cache sidecar 创建/删除均跳过,普通 profile.md 仍进入 DETAIL 索引
  • Ruff check、diff check、LoopX public-boundary scan 均通过

说明:tests/agfs/test_viking_fs_git.py -k classify_restore_path 在本机 macOS 因 AGFS 原生模块不可用而整模块 skip(未把它计作通过),提交后的 CI 正在覆盖。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

bug: memory overview cache key mismatch causes full LLM reprocessing on every semantic refresh

1 participant