Skip to content

feat(memory): post-final-answer memory extraction via standalone LLM call - #3682

Open
JasonW404 wants to merge 2 commits into
developfrom
feat/final-answer-based-memory
Open

feat(memory): post-final-answer memory extraction via standalone LLM call#3682
JasonW404 wants to merge 2 commits into
developfrom
feat/final-answer-based-memory

Conversation

@JasonW404

@JasonW404 JasonW404 commented Aug 17, 2026

Copy link
Copy Markdown
Member

Summary

Refactors the memory service execution logic to decouple memory extraction from the react loop.

Problem

Previously, memories were only stored when the LLM called StoreMemoryTool inline during the agent loop — a model-directed approach that was unreliable. The LLM often forgot or skipped the call at the final answer step.

Solution

Two complementary mechanisms:

1. Restrict StoreMemoryTool to action steps only

  • Updated system prompt (zh/en) to prohibit store_memory during the final answer step
  • Updated ToolConfig description to match the new policy
  • StoreMemoryTool now only fires during intermediate action steps for process-level observations

2. Post-final-answer extraction pipeline

  • New FaMemoryExtractor module (backend/services/fa_memory_extractor.py)
  • New extraction prompt YAML (backend/prompts/fa_memory_extraction_en.yaml)
  • Wired into agent_service.py finally block as fire-and-forget asyncio.create_task
  • Non-blocking — never delays the user response
  • Uses tenant LLM to parse <memory-item> tags from the final answer
  • Stores extracted items via MemoryService with layer=AGENT, memory_type=SHORT_TERM

3. Bug fix

  • memory_type enum normalization in _backend_store_hook used isinstance(x, MemoryLayer) instead of hasattr(x, 'value')

Files Changed

File Change
backend/services/fa_memory_extractor.py NewFaMemoryExtractor class + ExtractionResult dataclass
backend/prompts/fa_memory_extraction_en.yaml New — extraction prompt with <memory-item> output format
test/backend/services/test_fa_memory_extractor.py New — 29 unit tests
backend/utils/memory_tool_prompt.py Restrict store_memory to action steps
backend/agents/create_agent_info.py Update ToolConfig description
backend/services/memory_backend_adapter.py Add build_memory_service_for_fa_extraction() + fix enum bug
backend/services/agent_service.py Capture final answer + schedule extraction task
test/backend/services/test_memory_backend_adapter.py +9 tests (enum normalization, embedding skip, defaults)

Test Results

  • 29/29 new FaMemoryExtractor tests pass
  • 16/16 memory_backend_adapter tests pass (7 existing + 9 new)
  • 71/71 SDK memory tests pass (no regressions)
  • End-to-end verified through real chat API pipeline: 3 memories successfully extracted and stored from a single conversation
image image

…call

Decouple memory extraction from the react loop. Previously, memories were
only stored when the LLM called StoreMemoryTool inline — unreliable and
often missed. This refactor introduces two complementary mechanisms:

1. Restrict StoreMemoryTool to action steps only
   - Updated system prompt (zh/en) to prohibit store_memory during the
     final answer step
   - Updated ToolConfig description to match the new policy

2. Post-final-answer extraction pipeline
   - New FaMemoryExtractor module: validates, truncates, calls tenant
     LLM, parses <memory-item> tags, stores via MemoryService
   - New extraction prompt YAML with structured output format
   - Wired into agent_service.py finally block as fire-and-forget
     asyncio.create_task (non-blocking, never delays user response)
   - New build_memory_service_for_fa_extraction() factory in adapter

3. Bug fix: memory_type enum normalization in _backend_store_hook used
   isinstance(x, MemoryLayer) instead of hasattr(x, 'value')

4. Test coverage: 29 new FaMemoryExtractor tests + 9 new adapter tests
   (enum normalization, embedding skip, conversation_id coercion,
   search defaults, string layer serialization, minimal payload)
…only policy

The existing tests asserted on old prompt text ('Before every final answer',
'must call store_memory', '用户提问') that was deliberately removed in the
refactor. Updated assertions to match the new action-steps-only policy.
@JasonW404
JasonW404 marked this pull request as ready for review August 17, 2026 06:20
Copilot AI lite review requested due to automatic review settings August 17, 2026 06:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants