Backfill existing conversations & add monitoring for conversation memory

[lbedner]

Summary

Follow-up tasks after core conversation memory RAG is implemented.

Tasks

1. Migration: Backfill existing conversations

• One-off job to index all existing conversation messages into conversation-memory collection
• Should be idempotent (can re-run safely)
• CLI command: my-app ai memory backfill or similar

2. Monitoring: Add metrics for retrievals

• Track hits per query
• Track average similarity score
• Track retrieval latency
• Optional: dashboard in Overseer

3. QA: Integration test

• Add integration test that:
  1. Runs a short conversation
  2. References something from early in the conversation
  3. Verifies older facts are retrieved via RAG memory
  4. Confirms they appear in the response

Dependencies

• Requires Index conversation messages into conversation-memory collection #419, Retrieve relevant conversation history snippets before prompt construction #420, Add safe prompt framing for retrieved conversation memory #421 to be completed first

Estimate

1 day total