Persistent Memory System — AITEAM-X

Version: 3.0
Updated: 2026-03-24
Status: Implemented and in production

1. Overview

The memory system addresses context death: on each new session, AI agents lose all accumulated knowledge. The solution is a five-layer architecture that persists, structures, searches, injects, and compacts memory automatically.

┌──────────────────────────────────────────────────┐
│         Layer 5: Automatic Compaction             │
│  compact.ts cleans expired checkpoints,           │
│  trims conversations, consolidates vault, reindexes│
├──────────────────────────────────────────────────┤
│         Layer 4: Context Injection                │
│  inject.ts composes relevant memory               │
│  and injects it into new session prompts          │
├──────────────────────────────────────────────────┤
│         Layer 3: BM25 Search                        │
│  MiniSearch indexes the vault and retrieves       │
│  decisions/lessons relevant to the command        │
├──────────────────────────────────────────────────┤
│         Layer 2: Memory Vault                       │
│  Structured storage per agent and category        │
│  Markdown with ID and tags frontmatter            │
├──────────────────────────────────────────────────┤
│         Layer 1: Session Persistence                │
│  Chat sessions + conversation history             │
│  Checkpoints for recovery                         │
└──────────────────────────────────────────────────┘

Simplified data flow

flowchart LR
    UI["Frontend\n(MainContent)"] -->|auto-save 30s| CONV["Conversations\n(.memory/conversations/)"]
    UI -->|checkpoint 30s| CP["Checkpoints\n(.vault/checkpoints/)"]
    UI -->|close agent| HO["Handoff\n(vault/handoffs.md)"]
    HO --> VAULT["Memory Vault\n(.memory/{agentId}/)"]
    COMPACT["Compaction\n(compact.ts)"] -->|heuristic extraction| VAULT
    COMPACT -->|trims| CONV
    COMPACT -->|cleans| CP
    VAULT --> SEARCH["BM25 index\n(search.ts)"]
    SEARCH --> INJECT["Injection\n(inject.ts)"]
    VAULT --> INJECT
    INJECT -->|enriched prompt| AGENT["Cursor Agent CLI"]

2. Layer 1: Session Persistence

Chat Sessions (`docs/chat-sessions.json`)

Maps agentId → Cursor CLI chatId to resume sessions with --resume.

{ "bmad-master": "chat_abc123", "dev": "chat_def456" }

Conversation History (`.memory/conversations/`)

Auto-saved every 30s via setInterval in the frontend and on beforeunload via sendBeacon.

.memory/conversations/
├── bmad-master.json
├── dev.json
└── tech-writer.json

Format of each file:

{
  "agentId": "bmad-master",
  "savedAt": "2026-03-15T23:04:23.463Z",
  "messages": [
    { "role": "user", "text": "..." },
    { "role": "agent", "text": "..." }
  ]
}

Checkpoints (`.memory/.vault/checkpoints/{agentId}.json`)

Saved automatically every 30s by the frontend for each agent with an open bubble. Also saved via sendBeacon when closing the browser. Valid for 7 days (SEVEN_DAYS_MS = 7 * 24 * 3_600_000). Expired checkpoints are removed automatically by Layer 5 (Compaction).

{
  "agentId": "bmad-master",
  "savedAt": 1773679871839,
  "messages": [...],
  "chatId": "chat_abc123",
  "modelId": "claude-opus-4-6"
}

Each checkpoint stores the last 50 messages of the conversation.

Session API (`lib/memory/session.ts`)

checkpoint(agentId, messages, chatId?, modelId?)  // saves snapshot to .vault/checkpoints/
recover(agentId)                                   // reads checkpoint if < 7 days, null otherwise
sleep(agentId, messages, summary)                  // saves handoff to vault + checkpoint

recover flow:

Valid checkpoint (< 7 days) → InjectContext.recovering = true → injected into prompt
Expired or missing checkpoint → fresh session

3. Layer 2: Memory Vault

File Layout

.memory/
├── _project.md                    # Global project context (injected into all agents)
├── {agentId}/                     # Per-agent directory
│   ├── decisions.md               # Technical and architectural decisions
│   ├── lessons.md                 # Lessons learned and fixed bugs
│   ├── handoffs.md                # Session summaries (newest first)
│   ├── tasks.md                   # Open tasks (checkboxes [ ] / [x])
│   └── projects.md                # Per-agent project context
├── conversations/                 # Raw history (gitignored)
└── .vault/
    ├── index.json                 # Persisted BM25 MiniSearch index
    ├── compact-log.json           # Last compaction result
    └── checkpoints/{agentId}.json # Session checkpoints

Categories (`VaultCategory`)

Category	Use
`decisions`	Technical decisions ("we decided to use SSE", "we will adopt...")
`lessons`	Lessons learned, fixed bugs, insights ("I learned", "the problem was...")
`handoffs`	Summary of each session — generated by frontend when closing agent
`tasks`	Open tasks in `- [ ] description` format
`projects`	Project context (stack, architecture, goals)

Markdown Entry Format

<!-- id:1773679871839 -->
## 2026-03-16T16:51 · #react #typescript #components

Free-form memory content in markdown.

---

id is Date.now() ensuring uniqueness (monotonic bump on collision)
Tags extracted automatically via /#(\w+)/g from content
Entries sorted by descending id (newest first)
Tags #compacted and #auto-extract mark entries created by Layer 5 (compaction)

Vault API (`lib/memory/vault.ts`)

readCategory(agentId, category): Promise<VaultEntry[]>
appendEntry(agentId, category, content, tags?): Promise<VaultEntry>
updateEntry(agentId, category, id, content): Promise<void>
deleteEntry(agentId, category, id): Promise<void>
listAgents(): Promise<string[]>
getCategoryCounts(agentId): Promise<Record<VaultCategory, number>>

Write Serialization

All vault writes are serialized via writeQueue (Promise chain). This avoids race conditions when concurrent operations modify the same markdown file:

export let writeQueue: Promise<void> = Promise.resolve();

function enqueue<T>(fn: () => Promise<T>): Promise<T> {
  const result = writeQueue.then(fn);
  writeQueue = result.then(() => {}, () => {});
  return result;
}

A failed operation does not block the queue — the chain continues independently.

Search Index Updates

After each appendEntry and deleteEntry, the vault performs a dynamic import("./search") to update the BM25 index. Errors in that update are swallowed — the index is rebuilt on the next compaction.

4. Layer 3: BM25 Search (`lib/memory/search.ts`)

The MiniSearch index is built over all vault entries. Persisted in .memory/.vault/index.json to avoid rebuilding on every request.

Behavior

On first search: reads all vault files and builds the index
On later searches: loads index.json from disk
On new entry (appendEntry): updates index via updateIndex(entry)
On delete (deleteEntry): removes from index via removeFromIndex(id)

Search API

search(query, { agentId?, category?, limit? }): Promise<SearchResult[]>
updateIndex(entry: VaultEntry): Promise<void>
removeFromIndex(id: string): Promise<void>
buildIndex(): Promise<void>

interface SearchResult {
  entry: VaultEntry;
  score: number;
  snippet: string;  // ~120 chars with relevant excerpt
}

MiniSearch Configuration

Parameter	Value
Indexed fields	`content`, `tags`
Stored fields	`id`, `date`, `content`, `tags`, `agentId`, `category`
Fuzzy matching	`0.2`
Prefix search	`true`
Default limit	`10` results

Snippet Construction

The snippet is built by finding the first occurrence of a query word in the content. A window of ~30 characters before and 120 total characters is extracted for context.

5. Layer 4: Context Injection (`lib/memory/inject.ts`)

Called from POST /api/agents/command when starting a new chat session.

`buildContext(agentId, command)`

Builds InjectContext with a 2,000-token budget:

Source	Method	Limit
Global project context	`_project.md` (direct read)	no limit
Latest handoff	`readCategory(agentId, "handoffs")[0]`	1 entry
Relevant decisions	`search(command, { category: "decisions" })`	top 3
Relevant lessons	`search(command, { category: "lessons" })`	top 2
Open tasks	`readCategory(agentId, "tasks")` filtered by `[ ]`	all
Recovery snapshot	`recover(agentId)`	last 3 msgs

Token estimate: Math.ceil(text.length / 4) — simple heuristic from average chars/tokens ratio.

Token budget trimming (discard order if > 2,000 tokens):

Lessons dropped first
Decisions second
Handoff last (most valuable)

`buildTextBlock(ctx)`

Converts InjectContext into the text block injected into the prompt:

## MEMORY CONTEXT

Project:
[contents of _project.md]

Last Session:
[latest handoff]

Relevant Decisions:
- [relevant decision snippet]

Relevant Lessons:
- [relevant lesson snippet]

Open Tasks:
- [ ] open task

Recovering previous session:
[user]: ...
[agent]: ...

---

`buildMemoryInstructions(agentId)`

Generates instructions so the agent knows where to write memory on the filesystem:

## MEMORY: When asked to save/learn/remember, WRITE to files (don't just say you will).
Shared: .memory/_project.md | Personal: .memory/{agentId}/{decisions,lessons,tasks,handoffs}.md

`buildProjectScopeBlock(projectName, workspace)`

When AITEAM-X is installed inside another project, injects a scope block so agents analyze the host project, not the dashboard infrastructure.

Full flow in the agent pipeline

flowchart TD
    CMD["POST /api/agents/command"] --> CHAT{"chatId exists?"}
    CHAT -->|No| NEW["New session"]
    NEW --> BC["buildContext(agentId, command)"]
    BC --> BT["buildTextBlock(ctx)"]
    BT --> MDC["getAgentMdcContent()"]
    MDC --> PROMPT["prompt = persona + memory + command"]
    CHAT -->|Yes| RESUME["Existing session"]
    RESUME --> SIMPLE["prompt = user command only"]
    PROMPT --> SPAWN["spawn cursor-agent"]
    SIMPLE --> SPAWN
    SPAWN --> SSE["Stream SSE → frontend"]

6. Layer 5: Automatic Compaction (`lib/memory/compact.ts`)

Compaction addresses unbounded vault growth. An endpoint (POST /api/memory/compact) runs five sequential steps. The frontend triggers compaction automatically every 10 minutes.

Automatic trigger (frontend)

MainContent.tsx checks on mount whether the last compaction was more than 10 minutes ago. If so, it runs POST /api/memory/compact. A 10-minute setInterval keeps periodic compaction while the dashboard is open.

The five steps (Steps A–E)

flowchart TD
    START["POST /api/memory/compact"] --> A["Step A\ncleanStaleCheckpoints()"]
    A --> B["Step B\ntrimConversations()"]
    B --> C["Step C\ncapVaultEntries()"]
    C --> D["Step D\nrebuildSearchIndex()"]
    D --> E["Step E\ncleanLegacyFiles()"]
    E --> LOG["Persist compact-log.json"]

Step A: Expired checkpoint cleanup

Removes checkpoints in .memory/.vault/checkpoints/ older than 7 days. Corrupt checkpoints (invalid JSON) are also removed.

Parameter	Value
Threshold	7 days
Criterion	`Date.now() - checkpoint.savedAt > SEVEN_DAYS_MS`
Corrupt checkpoints	Removed unconditionally

Step B: Heuristic extraction and conversation trimming

For all conversations in .memory/conversations/, the system extracts insights via pattern matching. Conversations with more than 20 messages are additionally trimmed. A processed-conversations.json file tracks a hash per conversation to avoid redundant reprocessing.

For conversations with more than 20 messages:

Splits messages into "old" (removed) and "recent" (last 20 kept)
Extracts insights from old messages using pattern matching on agent lines:
- Decisions: regex (/\bdecid/i, /\bescolh/i, /\bwill use\b/i, etc.)
- Lessons: regex (/\blearned\b/i, /\bimportante/i, /\bdiscovery/i, etc.)
- Max 10 decisions and 10 lessons per trimmed conversation
- Each insight capped at 300 characters
- Only lines longer than 15 characters are analyzed
Persists insights to the vault via appendEntry() with tags ["compacted", "auto-extract"]
Generates a compacted handoff with the last 3 agent messages from the removed portion, tags ["compacted", "auto-handoff"]
Rewrites the conversation file with only the 20 most recent messages

Parameter	Value
MAX_CONVERSATION_MESSAGES	20
Decision patterns	10 regex (PT + EN)
Lesson patterns	10 regex (PT + EN)
Max insights per conversation	10 decisions + 10 lessons
Per-insight cap	300 chars
Minimum line length	15 chars

Decision regex:

/\bdecid/i, /\bchose\b/i, /\bwill use\b/i, /\bdecisão/i,
/\bescolh/i, /\boptamos/i, /\badotamos/i, /\bvamos usar\b/i,
/\bwent with\b/i, /\bsettled on\b/i

Lesson regex:

/\blearned\b/i, /\bimportant/i, /\bnote:/i, /\baprendemos/i,
/\bimportante/i, /\blição/i, /\bdiscovery/i, /\binsight/i,
/\bdescobr/i, /\bobserv/i

Step C: Vault category consolidation

For each agent and category, if entry count exceeds 30:

Keeps the 20 newest
Consolidates the rest into a single summary entry prefixed "Compacted N older entries:"
Each consolidated entry shown as - [date] preview (200 chars)
Rewrites the category file with 21 entries (20 originals + 1 summary)

Parameter	Value
MAX_VAULT_ENTRIES_PER_CATEGORY	30 (trigger)
KEEP_VAULT_ENTRIES	20 (retained)

Step D: Search index rebuild

Deletes index.json and calls buildIndex() via dynamic import of search.ts. Ensures consistency after trim/cap operations that mutate the vault directly.

Step E: Legacy file cleanup

Removes two file types:

.md.bak files: produced by migrate.ts during flat → vault migration
Flat agent .md files: removed when a corresponding vault directory already exists

Compaction API

`GET /api/memory/compact`

Returns the last compaction result (read from .vault/compact-log.json).

{
  "lastCompaction": {
    "timestamp": "2026-03-24T10:30:00.000Z",
    "checkpointsCleaned": 3,
    "conversationsTrimmed": 2,
    "vaultEntriesMerged": 15,
    "indexRebuilt": true,
    "legacyFilesCleaned": 1
  }
}

`POST /api/memory/compact`

Runs full compaction. Restricted to localhost (rejects requests with external x-forwarded-for or x-real-ip).

7. Handoff Generation

Handoffs are generated by the frontend (MainContent.tsx) when closing an agent chat window.

Close flow

sequenceDiagram
    participant U as User
    participant FE as MainContent.tsx
    participant CP as /api/memory/checkpoint
    participant VL as /api/memory/vault

    U->>FE: Close bubble (×) or drawer
    FE->>FE: Filter non-internal messages
    FE->>CP: POST checkpoint (agentId, messages, modelId)
    FE->>FE: saveHandoffToVault()
    FE->>FE: Format last 6 messages
    FE->>VL: POST entry (category: "handoffs", tags: auto-handoff, session-close)
    FE->>FE: Dismiss bubble/drawer

Automatic handoff format

saveHandoffToVault takes the last 6 messages, formats each as [User/Agent]: text (up to 200 chars), and saves as a vault entry with tags auto-handoff and session-close.

Insight extraction on chat close

Besides the handoff, the frontend also extracts decisions and lessons from the entire conversation using the same heuristic patterns as compaction (PT+EN regex: decidimos, escolhemos, will use, recomendação, stack principal, aprendemos, lição, risco, etc.). Extracted insights are saved as vault entries with tags auto-extract and session-close.

8. Flat → Vault Migration (`lib/memory/migrate.ts`)

Converts legacy flat per-agent .md memory files into the directory/category vault structure.

Flow

Scans .memory/ for .md files (excluding _ and . prefixes)
For each file: splits on ## Title sections
Classifies each section into a VaultCategory via SECTION_MAP (PT + EN, fuzzy match)
Persists each section as an entry via appendEntry()
Renames original file to .md.bak (idempotent: existing .bak or existing vault dir → skip)

Section mapping

Section (header)	Vault category
Decisões, Decisões Técnicas, Decisions	`decisions`
Aprendizados, Notas de Sessão, Lessons, Findings, Notes	`lessons`
Tarefas, Tasks	`tasks`
Contexto, Contexto de Projeto, Context, Projects	`projects`
Handoffs	`handoffs`
(fallback for any unrecognized section)	`lessons`

API

POST /api/memory/migrate — localhost only. Returns { migrated: string[], skipped: string[] }.

9. REST APIs

Route	Method	Description
`/api/memory`	GET	Load conversations (all or by agentId)
`/api/memory`	POST	Save conversations, memory append, init project
`/api/memory/vault`	GET	List agents, category counts, read entries
`/api/memory/vault`	POST	Create entry
`/api/memory/vault`	PUT	Update entry
`/api/memory/vault`	DELETE	Delete entry
`/api/memory/search`	GET	BM25 search in vault
`/api/memory/checkpoint`	GET	Recover session checkpoint
`/api/memory/checkpoint`	POST	Save session checkpoint
`/api/memory/compact`	GET	Last compaction result
`/api/memory/compact`	POST	Run full compaction (localhost only)
`/api/memory/migrate`	POST	Migrate flat `.md` to vault (localhost only)

`GET /api/memory/search` — Parameters

Param	Type	Required	Default
`q`	string	yes	—
`agentId`	string	no	all
`category`	VaultCategory	no	all
`limit`	number	no	10 (max 100)

`POST /api/memory/checkpoint` — Body

{
  "agentId": "bmad-master",
  "messages": [{ "role": "user", "text": "..." }],
  "chatId": "chat_abc123",
  "modelId": "claude-opus-4-6"
}

`POST /api/memory/vault` — Body

{
  "agentId": "bmad-master",
  "category": "decisions",
  "content": "We decided to use SSE instead of WebSockets.",
  "tags": ["sse", "architecture"]
}

10. TypeScript Types (`lib/memory/types.ts`)

type VaultCategory = "decisions" | "lessons" | "tasks" | "projects" | "handoffs";

const VAULT_CATEGORIES: VaultCategory[] = [
  "decisions", "lessons", "tasks", "projects", "handoffs",
];

interface ConversationMessage {
  role: "user" | "agent";
  text: string;
  internal?: boolean;   // internal messages not saved in checkpoints/handoffs
}

interface VaultEntry {
  id: string;           // Date.now().toString()
  date: string;         // ISO datetime "2026-03-16T14:32"
  content: string;      // free-form markdown
  tags: string[];       // extracted via /#(\w+)/g
  agentId: string;
  category: VaultCategory;
}

interface Checkpoint {
  agentId: string;
  savedAt: number;      // Date.now()
  messages: ConversationMessage[];
  chatId?: string;
  modelId?: string;
}

interface SearchResult {
  entry: VaultEntry;
  score: number;
  snippet: string;      // ~120 chars
}

interface InjectContext {
  projectContext: string;
  handoff?: string;
  decisions: SearchResult[];
  lessons: SearchResult[];
  tasks: string[];
  tokenEstimate: number;
  recovering?: boolean;
  recoverySnapshot?: ConversationMessage[];
}

interface CompactionResult {
  timestamp: string;
  checkpointsCleaned: number;
  conversationsTrimmed: number;
  vaultEntriesMerged: number;
  indexRebuilt: boolean;
  legacyFilesCleaned: number;
}

11. Module Dependency Graph

graph TD
    TYPES["types.ts"] --> VAULT["vault.ts"]
    TYPES --> SEARCH["search.ts"]
    TYPES --> SESSION["session.ts"]
    TYPES --> INJECT["inject.ts"]
    TYPES --> COMPACT["compact.ts"]
    TYPES --> MIGRATE["migrate.ts"]

    VAULT -->|readCategory, appendEntry| SEARCH
    VAULT -->|appendEntry| SESSION
    VAULT -->|readCategory, appendEntry, listAgents| COMPACT

    SEARCH -->|search| INJECT
    VAULT -->|readCategory| INJECT
    SESSION -->|recover| INJECT

    SEARCH -.->|dynamic import buildIndex| COMPACT
    VAULT -.->|dynamic import updateIndex/removeFromIndex| SEARCH

    style TYPES fill:#f9f,stroke:#333
    style VAULT fill:#bbf,stroke:#333
    style SEARCH fill:#bfb,stroke:#333
    style INJECT fill:#fbb,stroke:#333
    style COMPACT fill:#fbf,stroke:#333

Circular dependencies avoided via dynamic import():

vault.ts → search.ts: index update after append/delete (try/catch, errors swallowed)
compact.ts → search.ts: index rebuild after compaction

12. Filesystem Layout

.memory/
├── _project.md                         # Global context — injected into all agents
├── bmad-master/                        # bmad-master agent vault
│   ├── decisions.md                    #   Technical decisions
│   ├── lessons.md                      #   Lessons learned
│   ├── handoffs.md                     #   Session summaries
│   ├── tasks.md                        #   Open tasks
│   └── projects.md                     #   Project context
├── dev/                                # dev agent vault
│   └── ...
├── conversations/                      # Raw history (gitignored)
│   ├── bmad-master.json
│   └── dev.json
└── .vault/                             # Internal system data
    ├── index.json                      # BM25 index (MiniSearch)
    ├── compact-log.json                # Last compaction log
    └── checkpoints/                    # Session checkpoints
        ├── bmad-master.json
        └── dev.json

13. Gitignore and Versioning

Path	Version?	Reason
`.memory/_project.md`	Yes	Curated project context
`.memory/{agentId}/`	Yes	Structured vault with valuable memories
`.memory/.vault/index.json`	No	Rebuilt automatically
`.memory/.vault/compact-log.json`	No	Operational log, regenerated each compaction
`.memory/.vault/checkpoints/`	No	Volatile session data
`.memory/conversations/`	No	High volume, transient
`docs/chat-sessions.json`	No	Volatile session IDs

14. Error Handling Patterns

The system follows silent resilience: subsystem failures do not block the main flow.

Module	Failure scenario	Behavior
`vault.ts`	Dynamic import of search fails	Swallowed — index updated on next compaction
`vault.ts`	Category file missing	Returns empty array
`session.ts`	Corrupt checkpoint (invalid JSON)	Returns `null` — fresh session
`session.ts`	Expired checkpoint (> 7 days)	Returns `null`
`search.ts`	Missing `index.json`	Rebuilds index from scratch on next search
`compact.ts`	Corrupt conversation	Skips file, continues with others
`compact.ts`	Index rebuild fails	`indexRebuilt: false` in result
`compact.ts`	Legacy file delete fails	Swallowed via try/catch
`inject.ts`	BM25 search fails	Injected context without decisions/lessons

15. System Constants

Constant	Value	Module	Purpose
`TOKEN_BUDGET`	2000	inject.ts	Injected context token limit
`SEVEN_DAYS_MS`	604,800,000	session.ts, compact.ts	Checkpoint validity
`MAX_CONVERSATION_MESSAGES`	20	compact.ts	Conversation trim trigger
`MAX_VAULT_ENTRIES_PER_CATEGORY`	30	compact.ts	Consolidation trigger
`KEEP_VAULT_ENTRIES`	20	compact.ts	Entries kept after consolidation
`COMPACT_INTERVAL`	600,000 (10 min)	MainContent.tsx	Auto-compaction frequency
Checkpoint size	50 msgs	session.ts	Messages retained in checkpoint
Handoff preview	6 msgs	MainContent.tsx	Messages used for handoff
Search fuzzy	0.2	search.ts	Fuzzy matching tolerance
Search limit default	10	search.ts	Results per search

16. Utility Script

`scripts/import-conversations.mjs`

Imports existing conversation history (.memory/conversations/) into the vault using keyword matching.

Imports: handoffs, decisions (PT+EN keywords), lessons (PT+EN keywords), projects (_project.md)

node scripts/import-conversations.mjs

17. Planned Evolution

Phase	Feature	Priority	Status
v3.1	Heuristic insight extraction on session close and in compaction	High	Done
v3.2	LLM extraction (complementing heuristics)	Medium	Pending
v3.3	Semantic search with embeddings (in addition to BM25)	Medium	Pending
v3.4	Knowledge graph — wiki-links between agent memories	Low	Pending
v3.5	Context profiles — tune injection by task type	Low	Pending
v4.0	Sync with Obsidian vault	Low	Pending

Uh oh!

FilesExpand file tree

memory-system.md

Latest commit

History