feat: add project-scoped agent memory system#351
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 📝 WalkthroughWalkthroughIntegrates a memory subsystem: context loading now can include scored memory files producing a combined system prompt; memory usage is recorded after successful runs; a new learnings extraction flow parses agent output and appends learnings to persistent memory. Changes
Sequence Diagram(s)sequenceDiagram
participant Auto as AutoModeService
participant Agent as AgentService
participant Context as ContextLoader
participant Memory as MemoryLoader
participant Claude as Claude
participant FS as FileSystem
Auto->>Agent: trigger feature/pipeline
Agent->>Context: loadContextFiles({includeMemory:true, taskContext})
Context->>Memory: loadRelevantMemory(taskContext, maxMemoryFiles)
Memory->>FS: read .automaker/memory/*.md
Memory->>Memory: parse frontmatter, score & select top files
Memory-->>Context: memoryFiles + memoryPrompt
Context->>Context: combine context + memory => combinedSystemPrompt
Context-->>Agent: combinedSystemPrompt
Agent->>Claude: run feature with combinedSystemPrompt
Claude-->>Agent: feature output
alt success & memory loaded
Agent->>Memory: recordMemoryUsage(loadedFiles, output, success)
Memory->>FS: update usageStats
Agent->>Claude: request learnings extraction (structured JSON)
Claude-->>Agent: structured learnings
Agent->>Memory: appendLearning(learningEntry)
Memory->>FS: persist learning file/append
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested labels
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SuperComboGamer
added
the
Do Not Merge
Summary of ChangesHello @SuperComboGamer, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements a foundational project-scoped memory system for AI agents. Its primary goal is to enable agents to retain and leverage past learnings, thereby improving decision-making, reducing redundant efforts, and fostering project-specific institutional knowledge. The system is designed for seamless integration, automatically providing relevant historical context to agents and capturing new insights from their successful work. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a significant and well-designed feature: a project-scoped memory system for AI agents. The file-based approach is pragmatic and the overall architecture is sound. My review focuses on improving robustness, particularly around concurrent file access and data parsing. I've identified two high-severity race conditions in memory-loader.ts that could lead to data loss under concurrent execution. Additionally, there are several medium-severity issues related to fragile data parsing and type safety that should be addressed to make the system more resilient. Overall, this is a great addition, and with these refinements, it will be even more solid.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (7)
libs/utils/src/context-loader.ts (1)
236-246: Type assertion may mask interface mismatches.The cast
fsModule as MemoryFsModuleat line 242 could silently fail at runtime ifContextFsModuledoesn't include all methods required byMemoryFsModule(e.g.,writeFile,mkdir,appendFile). SincesecureFsis the default and implements both, this works in practice, but customfsModuleimplementations may not.Consider either:
- Extending
ContextFsModuleto include optional memory-related methods- Documenting this requirement in the JSDoc for
fsModuleapps/server/src/services/auto-mode-service.ts (3)
2828-2842: UseresolveModelString()for model resolution.Per coding guidelines, use
resolveModelString()from@automaker/model-resolverto convert model aliases to full model names instead of directly accessingCLAUDE_MODEL_MAP.haiku.🔎 Proposed fix
// Import query dynamically to avoid circular dependencies const { query } = await import('@anthropic-ai/claude-agent-sdk'); - const { CLAUDE_MODEL_MAP } = await import('@automaker/model-resolver'); + const { resolveModelString, DEFAULT_MODELS } = await import('@automaker/model-resolver'); // Use a quick model for extraction const stream = query({ prompt: userPrompt, options: { - model: CLAUDE_MODEL_MAP.haiku, + model: resolveModelString('haiku', DEFAULT_MODELS.haiku), maxTurns: 1, allowedTools: [], permissionMode: 'acceptEdits', }, });
2859-2863: Add validation for untrusted JSON structure.The parsed JSON from Claude's response is used directly without validating array element structure beyond
learning.categoryandlearning.content. Consider adding type guards or validation to ensure the structure matchesLearningEntrybefore passing toappendLearning.🔎 Proposed validation
// Parse the response const jsonMatch = responseText.match(/\{[\s\S]*"learnings"[\s\S]*\}/); if (!jsonMatch) return; const parsed = JSON.parse(jsonMatch[0]); if (!parsed.learnings || !Array.isArray(parsed.learnings)) return; // Record each learning for (const learning of parsed.learnings) { - if (!learning.category || !learning.content) continue; + // Validate learning structure matches LearningEntry + if (!learning.category || typeof learning.category !== 'string') continue; + if (!learning.content || typeof learning.content !== 'string') continue; + const validTypes = ['decision', 'learning', 'pattern', 'gotcha']; + const type = validTypes.includes(learning.type) ? learning.type : 'learning'; await appendLearning( projectPath, { category: learning.category, - type: learning.type || 'learning', + type, content: learning.content, - why: learning.why, - rejected: learning.rejected, - breaking: learning.breaking, + why: typeof learning.why === 'string' ? learning.why : undefined, + rejected: typeof learning.rejected === 'string' ? learning.rejected : undefined, + breaking: typeof learning.breaking === 'string' ? learning.breaking : undefined, }, secureFs as Parameters<typeof appendLearning>[2] );
2794-2800: Consider making the minimum output length configurable.The hardcoded threshold of 100 characters and 10000 character truncation limit may not suit all use cases. These could be constants at the top of the file for easier tuning.
libs/utils/src/memory-loader.ts (3)
188-250: Consider externalizing stopWords for configurability.The stopWords list includes domain-specific terms (
add,create,implement, etc.) mixed with natural language stop words. Consider:
- Separating these into two lists for clarity
- Making this configurable per-project in the future
260-272: Usage score calculation may benefit from bounds clamping.
calculateUsageScorecan theoretically return values above 1.0 (0.5 + 0.3 + 0.2 = 1.0 max in practice, but edge cases with floating point). Consider addingMath.min(result, 1.0)for consistency if this is meant to be a 0-1 range.
103-164: Custom YAML parsing has real fragility for multi-line summaries and special characters in arrays.The regex-based parsing works for the intended simple format but has genuine limitations:
- Multi-line summaries will not parse (regex
/summary:\s*(.+)/doesn't match across newlines)- Special characters like commas within tag or array values will break the parsing logic
- Format extension beyond the current fields would require parser changes
This is acceptable for your controlled format, but these constraints should be documented at the function level for future developers. Consider adding JSDoc comments specifying the expected format structure.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
apps/server/src/services/auto-mode-service.tslibs/utils/src/context-loader.tslibs/utils/src/index.tslibs/utils/src/memory-loader.ts
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Always import from shared packages (@automaker/*), never from old relative paths
Files:
libs/utils/src/index.tslibs/utils/src/context-loader.tslibs/utils/src/memory-loader.tsapps/server/src/services/auto-mode-service.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
resolveModelString()from @automaker/model-resolver to convert model aliases (haiku, sonnet, opus) to full model names
Files:
libs/utils/src/index.tslibs/utils/src/context-loader.tslibs/utils/src/memory-loader.tsapps/server/src/services/auto-mode-service.ts
apps/server/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
createEventEmitter()fromlib/events.tsfor all server operations to emit events that stream to frontend via WebSocket
Files:
apps/server/src/services/auto-mode-service.ts
🧠 Learnings (1)
📚 Learning: 2025-12-28T05:07:48.147Z
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Store project-specific rules in `.automaker/context/` and load them into agent prompts via `loadContextFiles()` from automaker/utils
Applied to files:
libs/utils/src/context-loader.tslibs/utils/src/memory-loader.tsapps/server/src/services/auto-mode-service.ts
🧬 Code graph analysis (1)
libs/utils/src/context-loader.ts (1)
libs/utils/src/memory-loader.ts (4)
getMemoryDir(95-97)initializeMemoryFolder(521-592)MemoryFsModule(17-24)parseFrontmatter(103-164)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: e2e
🔇 Additional comments (14)
libs/utils/src/context-loader.ts (4)
18-23: LGTM - Memory integration imports are well-organized.The imports from
./memory-loader.jsare appropriate and include only what's needed for memory functionality integration.
43-51: LGTM - Clean interface definition.The
MemoryFileInfointerface is well-structured with appropriate fields for memory file tracking.
283-299: LGTM - Combined prompt building is well-structured.The logic correctly combines context and memory prompts with proper filtering of empty strings. The logging provides good observability.
302-327: LGTM - Memory prompt formatting is clear and consistent.The
buildMemoryPromptfunction follows the same pattern asbuildContextPromptand produces well-formatted output for the system prompt.libs/utils/src/index.ts (1)
59-82: LGTM - Public API exports are comprehensive and well-organized.The memory-related exports follow the existing pattern in this file and provide a clean public API surface for consumers.
apps/server/src/services/auto-mode-service.ts (3)
14-21: LGTM - Imports follow coding guidelines.The imports from
@automaker/utilsfollow the coding guideline to always import from shared packages (@automaker/*).
511-521: LGTM - Context and memory loading integration.The combined system prompt now properly includes both context files and memory files. The comment at line 520 correctly documents that
formattedPromptincludes both.
598-626: LGTM - Memory usage recording has appropriate error handling.The try-catch wrapper ensures that failures in learning extraction don't break the main feature completion flow. The conditional check for
memoryFiles.length > 0 && agentOutputprevents unnecessary work.libs/utils/src/memory-loader.ts (6)
1-24: LGTM - Well-documented module with clear interface.The module documentation and
MemoryFsModuleinterface are well-structured and compatible with the secureFs pattern used elsewhere.
286-369: LGTM - Relevance scoring logic is well-designed.The scoring approach with weighted tag/relevantTo/summary matching, combined with usage and importance multipliers, is a reasonable heuristic. Always including
gotchas.mdis a good safety feature.
429-454: Heuristic for "referenced" detection may have false positives.Using term matching (
countMatches >= 3) to determine if a file was referenced is a rough heuristic. Common terms in memory files may match agent output coincidentally. Consider:
- Requiring a higher threshold
- Using more specific markers or content signatures
This is acceptable for v1 but worth monitoring.
485-515: LGTM - Learning persistence with proper file creation.The
appendLearningfunction correctly handles both existing files (append) and new files (create with frontmatter). The filename sanitization is appropriate.
521-591: LGTM - Memory folder initialization is idempotent.The function correctly checks for existing directory before creating, and sets up sensible defaults for
_index.mdandgotchas.md.
139-143: Typo in regex prevents importance from being parsed.There's a typo on line 140:
import anceshould beimportance. This bug prevents the importance value from being parsed from frontmatter.🔎 Proposed fix
// Parse importance - const importanceMatch = frontmatterStr.match(/import ance:\s*([\d.]+)/); + const importanceMatch = frontmatterStr.match(/importance:\s*([\d.]+)/); if (importanceMatch) { metadata.importance = parseFloat(importanceMatch[1]); }Likely an incorrect or invalid review comment.
- Add taskContext parameter to loadContextFiles for intelligent file selection - Memory files are scored based on tag matching with task keywords - Category name matching (e.g., "terminals" matches terminals.md) with 4x weight - Usage statistics influence scoring (files that helped before rank higher) - Limit to top 5 files + always include gotchas.md - Auto-mode passes feature title/description as context - Chat sessions pass user message as context This prevents loading 40+ memory files and killing context limits. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
apps/server/src/services/auto-mode-service.ts (2)
2863-2867: JSON extraction regex is fragile.The regex
/\{[\s\S]*"learnings"[\s\S]*\}/is greedy and may match unintended content if the model outputs additional JSON or curly braces. Consider extracting JSON code blocks first (json...) as a more robust approach.🔎 Suggested improvement
- const jsonMatch = responseText.match(/\{[\s\S]*"learnings"[\s\S]*\}/); - if (!jsonMatch) return; - - const parsed = JSON.parse(jsonMatch[0]); + // Try to extract JSON from code block first, then fall back to raw JSON + const codeBlockMatch = responseText.match(/```(?:json)?\s*(\{[\s\S]*?\})\s*```/); + const jsonMatch = codeBlockMatch?.[1] || responseText.match(/\{[\s\S]*?"learnings"\s*:\s*\[[\s\S]*?\]\s*\}/)?.[0]; + if (!jsonMatch) return; + + const parsed = JSON.parse(jsonMatch);
2870-2871: Category values from LLM are not validated.The
learning.categoryreceived from the model is passed directly toappendLearningwithout validation. If the model hallucinates a category, it will create arbitrary memory files. Consider validating against the expected categories listed in the prompt.🔎 Suggested fix with allowlist validation
+ const allowedCategories = new Set([ + 'architecture', 'api', 'ui', 'terminals', 'auth', 'testing', 'gotchas' + ]); + // Record each learning for (const learning of parsed.learnings) { - if (!learning.category || !learning.content) continue; + if (!learning.category || !learning.content) continue; + // Skip invalid categories to prevent arbitrary file creation + if (!allowedCategories.has(learning.category)) { + console.warn(`[AutoMode] Skipping learning with invalid category: ${learning.category}`); + continue; + }libs/utils/src/context-loader.ts (1)
92-92: Type safety concern with fsModule parameter.The
fsModuleis typed asContextFsModule(read-only operations), butinitializeMemoryFolderon line 261 requires write operations. The unsafe castfsModule as MemoryFsModulecan cause runtime errors if a consumer passes a read-only fs module.Consider updating the type signature to explicitly require write permissions when
initializeMemoryis true, or document this requirement clearly.
🧹 Nitpick comments (1)
libs/utils/src/context-loader.ts (1)
344-349: High-importance file selection may exceed maxMemoryFiles.The loop adds high-importance files (importance >= 0.9) but only checks
selectedFiles.size < maxMemoryFilesafter the check for importance. If there are many high-importance files, this condition allows selecting up tomaxMemoryFiles, which is correct. However,gotchas.mdis added unconditionally before this loop, so if there are exactlymaxMemoryFileshigh-importance files, the total could bemaxMemoryFiles + 1.Consider whether
gotchas.mdshould count toward the limit:// Always include gotchas.md const gotchasFile = scoredFiles.find((f) => f.fileName === 'gotchas.md'); if (gotchasFile) { selectedFiles.add('gotchas.md'); } // Add high-importance files for (const file of scoredFiles) { - if (file.metadata.importance >= 0.9 && selectedFiles.size < maxMemoryFiles) { + if (file.metadata.importance >= 0.9 && !selectedFiles.has(file.fileName) && selectedFiles.size < maxMemoryFiles) { selectedFiles.add(file.fileName); } }
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
apps/server/src/services/agent-service.tsapps/server/src/services/auto-mode-service.tslibs/utils/src/context-loader.tslibs/utils/src/index.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- libs/utils/src/index.ts
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Always import from shared packages (@automaker/*), never from old relative paths
Files:
apps/server/src/services/agent-service.tslibs/utils/src/context-loader.tsapps/server/src/services/auto-mode-service.ts
apps/server/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
createEventEmitter()fromlib/events.tsfor all server operations to emit events that stream to frontend via WebSocket
Files:
apps/server/src/services/agent-service.tsapps/server/src/services/auto-mode-service.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
resolveModelString()from @automaker/model-resolver to convert model aliases (haiku, sonnet, opus) to full model names
Files:
apps/server/src/services/agent-service.tslibs/utils/src/context-loader.tsapps/server/src/services/auto-mode-service.ts
🧠 Learnings (2)
📓 Common learnings
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Store project-specific rules in `.automaker/context/` and load them into agent prompts via `loadContextFiles()` from automaker/utils
📚 Learning: 2025-12-28T05:07:48.147Z
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Store project-specific rules in `.automaker/context/` and load them into agent prompts via `loadContextFiles()` from automaker/utils
Applied to files:
apps/server/src/services/agent-service.tslibs/utils/src/context-loader.tsapps/server/src/services/auto-mode-service.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: e2e
🔇 Additional comments (7)
apps/server/src/services/agent-service.ts (1)
241-250: LGTM! Memory-aware context loading properly integrated.The
taskContextconstruction using the user message for smart memory selection is well-designed. Using the first 200 characters as the title and the full message as the description provides a good balance for memory file scoring.apps/server/src/services/auto-mode-service.ts (3)
19-20: Imports follow shared package pattern correctly.The new memory-related imports (
appendLearning,recordMemoryUsage) are properly imported from@automaker/utils, following the coding guidelines.
511-525: Context and memory integration properly implemented.The
taskContextis correctly constructed from feature metadata, and the combined system prompt now includes both context and memory files. This aligns with the memory-aware loading pattern used in agent-service.
602-631: Memory usage recording and learnings extraction well-implemented.The post-success handling appropriately:
- Reads agent output with a graceful fallback if missing
- Only records memory usage when memory files were actually loaded
- Wraps learning extraction in try-catch to avoid failing successful features
libs/utils/src/context-loader.ts (3)
200-252: Context file loading logic is well-structured.The refactored code cleanly separates context and memory file arrays, with appropriate error handling and logging.
381-398: Combined prompt construction and logging are well-implemented.The prompt building correctly combines context and memory prompts, filtering out empty strings. The logging provides useful visibility into what was loaded without being overly verbose.
408-433: Memory prompt formatting is clear and consistent.The
buildMemoryPromptfunction follows the same formatting pattern asbuildContextPrompt, with appropriate headers and separators. The emphasis on reviewing past decisions before making changes is a good UX touch.
- Improved context loading by adding task context for better memory selection. - Updated JSON parsing logic to handle various formats and ensure robust error handling. - Introduced file locking mechanisms to prevent race conditions during memory file updates. - Enhanced metadata handling in memory files, including validation and sanitization. - Refactored scoring logic for context files to improve selection accuracy based on task relevance. These changes optimize memory file management and enhance the overall performance of the auto-mode service.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (5)
libs/utils/src/memory-loader.ts (2)
461-480: LGTM - Race condition inincrementUsageStatis now addressed.The function correctly uses
withFileLockto serialize read-modify-write operations, resolving the race condition flagged in a previous review.
551-590: LGTM - TOCTOU race inappendLearningis now addressed.The entire check-then-act sequence (access check → create or append) is now wrapped in
withFileLock, eliminating the TOCTOU race condition flagged in the previous review.apps/server/src/services/auto-mode-service.ts (2)
2845-2853: LGTM - Correctly usesresolveModelString('haiku')per coding guidelines.This addresses the previous review comment about using
CLAUDE_MODEL_MAP.haikudirectly. The code now properly uses the model resolver function as required.
2915-2937: Category validation is missing - LLM may create arbitrary memory files.A previous review suggested validating categories against an allowlist (
architecture,api,ui,terminals,auth,testing,gotchas). The current code only validatestypebut accepts anycategorystring, which could lead to memory file proliferation if the LLM hallucinates categories.🔎 Suggested fix to add category validation
// Valid learning types const validTypes = new Set(['decision', 'learning', 'pattern', 'gotcha']); + // Valid categories to prevent file proliferation + const validCategories = new Set([ + 'architecture', 'api', 'ui', 'terminals', 'auth', 'testing', 'gotchas', 'general' + ]); // Record each learning for (const item of parsed.learnings) { // Validate required fields with proper type narrowing if (!item || typeof item !== 'object') continue; const learning = item as Record<string, unknown>; if ( !learning.category || typeof learning.category !== 'string' || !learning.content || typeof learning.content !== 'string' || !learning.content.trim() ) { continue; } + // Normalize category - map to 'general' if not in allowlist + const category = validCategories.has(learning.category) + ? learning.category + : 'general'; // Validate and normalize type const typeStr = typeof learning.type === 'string' ? learning.type : 'learning'; const learningType = validTypes.has(typeStr) ? (typeStr as 'decision' | 'learning' | 'pattern' | 'gotcha') : 'learning'; await appendLearning( projectPath, { - category: learning.category, + category, type: learningType,libs/utils/src/context-loader.ts (1)
72-80: LGTM -ContextFsModulenow correctly declares write methods.This addresses the previous review comment about the type mismatch. The interface now includes
writeFile,mkdir, andappendFile, making it compatible with memory operations that require write access.
🧹 Nitpick comments (4)
libs/utils/src/memory-loader.ts (2)
151-207: Regex-based YAML parsing is fragile but acceptable with current safeguards.The past review flagged this as fragile. While a proper YAML library (e.g.,
js-yaml) would be more robust, the current implementation:
- Has safe fallback to
createDefaultMetadata()on parse failure- Clamps importance to valid range (0-1)
- Filters empty strings from arrays
This is acceptable for the controlled frontmatter format, but be aware that edge cases (e.g., colons in tag values) may silently fall back to defaults. Consider adding a comment documenting the expected format.
506-518: Heuristic reference detection may produce false positives.The logic considers a file "referenced" if ≥3 terms from the memory file appear in the agent output. Common terms (e.g., "function", "component", "data") could trigger false matches, inflating
referencedandsuccessfulFeaturesstats over time.Consider:
- Using a higher threshold (e.g., 5+ matches)
- Weighting by term rarity or requiring phrase matches
- Documenting this as an approximation that may need tuning
libs/utils/src/context-loader.ts (2)
289-340: Memory scoring logic is duplicated frommemory-loader.ts.The scoring calculation here (lines 307-330) largely duplicates the logic in
loadRelevantMemory()frommemory-loader.ts(lines 377-387), with the addition ofcategoryScore. This could lead to divergence if one is updated but not the other.Consider refactoring to share a common scoring function, or document why the two implementations intentionally differ (e.g.,
categoryScoreadded here).🔎 Suggested approach
Extract a shared
calculateRelevanceScorefunction inmemory-loader.ts:export function calculateRelevanceScore( metadata: MemoryMetadata, taskTerms: string[], categoryTerms?: string[] ): number { const tagScore = countMatches(metadata.tags, taskTerms) * 3; const relevantToScore = countMatches(metadata.relevantTo, taskTerms) * 2; const summaryTerms = extractTerms(metadata.summary); const summaryScore = countMatches(summaryTerms, taskTerms); const categoryScore = categoryTerms ? countMatches(categoryTerms, taskTerms) * 4 : 0; const usageScore = calculateUsageScore(metadata.usageStats); return (tagScore + relevantToScore + summaryScore + categoryScore) * metadata.importance * usageScore; }Then use it in both
loadRelevantMemoryandloadContextFiles.
427-449:buildMemoryPromptduplicates functionality frommemory-loader.ts.This function (lines 427-449) is nearly identical to
buildMemoryPromptinmemory-loader.ts(lines 435-455). The only difference is formatting the category name.Consider exporting and reusing the one from
memory-loader.ts, or consolidating to avoid divergence.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
apps/server/src/services/auto-mode-service.tslibs/utils/src/context-loader.tslibs/utils/src/index.tslibs/utils/src/memory-loader.ts
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Always import from shared packages (@automaker/*), never from old relative paths
Files:
apps/server/src/services/auto-mode-service.tslibs/utils/src/context-loader.tslibs/utils/src/index.tslibs/utils/src/memory-loader.ts
apps/server/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
createEventEmitter()fromlib/events.tsfor all server operations to emit events that stream to frontend via WebSocket
Files:
apps/server/src/services/auto-mode-service.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
resolveModelString()from @automaker/model-resolver to convert model aliases (haiku, sonnet, opus) to full model names
Files:
apps/server/src/services/auto-mode-service.tslibs/utils/src/context-loader.tslibs/utils/src/index.tslibs/utils/src/memory-loader.ts
🧠 Learnings (2)
📚 Learning: 2025-12-28T05:07:48.147Z
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Applies to **/*.{ts,tsx} : Use `resolveModelString()` from automaker/model-resolver to convert model aliases (haiku, sonnet, opus) to full model names
Applied to files:
apps/server/src/services/auto-mode-service.ts
📚 Learning: 2025-12-28T05:07:48.147Z
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Store project-specific rules in `.automaker/context/` and load them into agent prompts via `loadContextFiles()` from automaker/utils
Applied to files:
apps/server/src/services/auto-mode-service.tslibs/utils/src/context-loader.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: e2e
🔇 Additional comments (10)
libs/utils/src/memory-loader.ts (3)
214-221: LGTM - YAML string escaping addresses the previous review concern.The
escapeYamlStringfunction now properly handles special YAML characters and is applied to all string fields inserializeFrontmatter. This resolves the past issue about unquoted summary values.
596-667: LGTM - Memory folder initialization is well-structured.The initialization creates a sensible default structure with
_index.mdfor documentation andgotchas.mdwith high importance (0.9). The idempotent check prevents re-initialization if the folder already exists.
96-123: In-memory file locking is appropriate for the current single-process deployment model.The
fileLocksMap correctly prevents race conditions within a single Node.js process instance. Since the application runs as a single process per container (with no clustering, worker threads, or multi-process patterns),withFileLockis sufficient for synchronizing access to shared files like the memory store and learnings file.If the application scales horizontally in the future with multiple container instances accessing the same shared file storage, consider upgrading to file-system-level locks (e.g.,
proper-lockfile) or an atomic write pattern.apps/server/src/services/auto-mode-service.ts (4)
19-21: LGTM - Clean import additions for memory functionality.The new imports from
@automaker/utilsfollow the coding guideline to import from shared packages rather than relative paths.
513-525: LGTM - Task context properly passed for smart memory selection.The
taskContextwith feature title and description enables the memory loader to score and select relevant memory files based on the current task.
602-631: Good integration of memory recording after successful feature completion.The code correctly:
- Only records memory usage when memory files were loaded
- Wraps learning extraction in try/catch to prevent failures from blocking feature completion
- Uses the agent output for context
2870-2900: Improved JSON extraction with code block detection and balanced brace finding.This addresses the previous review concern about the greedy regex. The new approach:
- First tries to extract from markdown code blocks (common LLM pattern)
- Falls back to balanced brace matching starting from
"learnings"This is more robust than the original greedy regex.
libs/utils/src/index.ts (1)
52-85: LGTM - Clean export additions for the new memory system.The new exports follow the existing barrel export pattern and properly expose:
- Context-related types (
ContextFsModule,MemoryFileInfo,TaskContext)- Memory functions (
loadRelevantMemory,appendLearning,recordMemoryUsage, etc.)- Memory types (
MemoryMetadata,MemoryFile,LearningEntry, etc.)This enables downstream modules to access the memory system through the unified
@automaker/utilspackage.libs/utils/src/context-loader.ts (2)
386-393: Incrementingloadedstat during context load is correct but increases write operations.Each selected memory file triggers a write to update its
loadedstat. For 5 memory files, this means 5 additional file writes per context load. This is acceptable for the current use case but could become a bottleneck at scale.The non-critical error handling (silent catch) is appropriate here.
207-422: Overall memory integration inloadContextFilesis well-structured.The implementation:
- Properly defaults
includeMemoryandinitializeMemorytotrue- Uses smart scoring when
taskContextis provided, falls back to importance-only otherwise- Always includes
gotchas.mdand high-importance files- Respects
maxMemoryFileslimit- Combines context and memory prompts cleanly
…vice - Improved the learning extraction process by refining the user prompt to focus on meaningful insights and structured JSON output. - Updated the LearningEntry interface to include additional context fields for better documentation of decisions and patterns. - Enhanced the formatLearning function to adopt an Architecture Decision Record (ADR) style, providing richer context for recorded learnings. - Added detailed logging for better traceability during the learning extraction and appending processes. These changes aim to improve the quality and clarity of learnings captured during the auto-mode service's operation.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
libs/utils/src/memory-loader.ts (1)
154-211: Regex-based YAML parsing is fragile for edge cases.The custom YAML parsing can fail with colons in summaries, multi-line values, or unusual formatting. Since these files may be manually edited, consider using a lightweight YAML library like
js-yamlfor robustness.apps/server/src/services/auto-mode-service.ts (1)
2942-2979: Consider validating learning categories against an allowlist.The
learning.categoryfrom LLM output is used directly without validation. If the model hallucinates categories, this could create many unwanted memory files. Consider validating against expected categories (architecture, api, ui, database, auth, testing, performance, security, gotchas).🔎 Proposed fix to validate categories
+ // Valid categories to prevent file proliferation from hallucinated categories + const validCategories = new Set([ + 'architecture', 'api', 'ui', 'database', 'auth', + 'testing', 'performance', 'security', 'gotchas', 'general' + ]); + // Record each learning for (const item of parsed.learnings) { // Validate required fields with proper type narrowing if (!item || typeof item !== 'object') continue; const learning = item as Record<string, unknown>; if ( !learning.category || typeof learning.category !== 'string' || !learning.content || typeof learning.content !== 'string' || !learning.content.trim() ) { continue; } + + // Normalize category to valid value, default to 'general' + const normalizedCategory = validCategories.has(learning.category.toLowerCase()) + ? learning.category.toLowerCase() + : 'general';
🧹 Nitpick comments (2)
libs/utils/src/memory-loader.ts (2)
98-126: In-memory lock doesn't protect against concurrent processes.The
withFileLockimplementation uses an in-memoryMapwhich only works within a single Node.js process. If multiple server instances or worker processes access the same memory files, the lock provides no protection against concurrent writes. This is acceptable for single-instance deployments but worth documenting.For multi-process safety, consider using a file-based lock (e.g.,
proper-lockfilepackage) or advisory locks.
380-394: Scoring weights differ from PR description.The PR objectives mention "category name 4x" as a scoring factor, but the implementation only scores tags (3x), relevantTo (2x), and summary (1x). If category-based matching was intended (matching file name against feature terms), consider adding it. Otherwise, update documentation.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
apps/server/src/services/auto-mode-service.tslibs/utils/src/memory-loader.ts
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Always import from shared packages (@automaker/*), never from old relative paths
Files:
apps/server/src/services/auto-mode-service.tslibs/utils/src/memory-loader.ts
apps/server/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
createEventEmitter()fromlib/events.tsfor all server operations to emit events that stream to frontend via WebSocket
Files:
apps/server/src/services/auto-mode-service.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
resolveModelString()from @automaker/model-resolver to convert model aliases (haiku, sonnet, opus) to full model names
Files:
apps/server/src/services/auto-mode-service.tslibs/utils/src/memory-loader.ts
🧠 Learnings (3)
📓 Common learnings
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Store project-specific rules in `.automaker/context/` and load them into agent prompts via `loadContextFiles()` from automaker/utils
📚 Learning: 2025-12-28T05:07:48.147Z
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Applies to **/*.{ts,tsx} : Use `resolveModelString()` from automaker/model-resolver to convert model aliases (haiku, sonnet, opus) to full model names
Applied to files:
apps/server/src/services/auto-mode-service.ts
📚 Learning: 2025-12-28T05:07:48.147Z
Learnt from: CR
Repo: AutoMaker-Org/automaker PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-28T05:07:48.147Z
Learning: Store project-specific rules in `.automaker/context/` and load them into agent prompts via `loadContextFiles()` from automaker/utils
Applied to files:
apps/server/src/services/auto-mode-service.ts
🧬 Code graph analysis (1)
apps/server/src/services/auto-mode-service.ts (3)
libs/utils/src/context-loader.ts (1)
loadContextFiles(207-422)apps/server/src/lib/settings-helpers.ts (1)
filterClaudeMdFromContext(98-137)libs/utils/src/memory-loader.ts (2)
recordMemoryUsage(497-522)appendLearning(564-608)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: e2e
🔇 Additional comments (17)
libs/utils/src/memory-loader.ts (9)
213-246: LGTM - YAML escaping properly handles special characters.The
escapeYamlStringhelper addresses the concern about special characters in summaries and other fields by detecting problematic characters and wrapping values in escaped double quotes.
248-314: LGTM - Term extraction with sensible stop words.The stop word list includes both common English words and generic action verbs, which helps focus matching on meaningful feature-specific terms.
324-336: LGTM - Usage score calculation handles edge cases.Properly guards against division by zero and returns a neutral score for new files.
404-427: LGTM - Always includes gotchas.md with proper fallback.The logic correctly ensures
gotchas.mdis always loaded, even if it wasn't in the scored results due to low relevance score.
464-483: Race condition addressed with file locking.The function now uses
withFileLockto protect the read-modify-write operation, addressing the previously flagged concern for single-process scenarios.
497-522: LGTM - Heuristic-based reference detection.The term-matching approach with a threshold of 3 provides a reasonable heuristic for determining whether memory content was actually used by the agent.
528-557: LGTM - Clean ADR-style formatting.The function produces well-structured markdown entries with appropriate headers and optional field handling for different learning types.
564-608: TOCTOU race condition addressed with file locking.The function now wraps the check-then-write logic in
withFileLock, addressing the previously flagged race condition. Category sanitization also prevents path traversal issues.
614-685: LGTM - Memory folder initialization.Creates the memory directory with helpful starter files. The early return when the directory exists prevents unnecessary operations.
apps/server/src/services/auto-mode-service.ts (8)
18-21: LGTM - Imports follow shared package pattern.New memory utilities are correctly imported from
@automaker/utilsas per coding guidelines.
511-525: LGTM - Task context enables smart memory selection.Passing feature title and description to
loadContextFilesallows the memory loader to select relevant memory files based on the current task.
602-630: LGTM - Memory tracking with graceful error handling.Memory usage recording and learning extraction are wrapped in try-catch to prevent failures from breaking the feature completion flow. The checks for non-empty memory files and agent output are appropriate.
698-707: LGTM - Consistent task context for pipeline steps.Pipeline steps use the same pattern for loading context with task-aware memory selection.
937-940: LGTM - Task context with prompt fallback.When feature data isn't available, falls back to using the prompt content for memory selection.
2853-2863: LGTM - UsesresolveModelStringas required.The model alias 'haiku' is correctly resolved using
resolveModelString('haiku')per coding guidelines, replacing the previous directCLAUDE_MODEL_MAPaccess.
2883-2919: JSON extraction improved with code block handling.The implementation now first tries to extract JSON from markdown code blocks before falling back to balanced brace matching, which is more robust than the previous greedy regex approach.
2806-2812: LGTM - Early exit for insufficient output.Skipping learning extraction for outputs under 100 characters prevents wasted API calls when there's nothing meaningful to extract.
SuperComboGamer
added
the
Testers-Requested
- Added stripProviderPrefix utility to various routes to ensure providers receive bare model IDs. - Updated model references in executeQuery calls across multiple files, enhancing consistency in model ID handling. - Introduced memoryExtractionModel in settings for improved learning extraction tasks. These changes streamline the model ID processing and enhance the overall functionality of the provider interactions.
…ctions - Improved error handling in the handleRunFeature and handleStartImplementation functions to throw errors for better caller management. - Integrated connection error detection and server offline handling, redirecting users to the login page when the server is unreachable. - Updated follow-up feature logic to include rollback mechanisms and improved user feedback for error scenarios. These changes enhance the robustness of the board actions by ensuring proper error management and user experience during server connectivity issues.
2 participants
Summary
Adds a project-scoped memory system that allows AI agents to learn from past work and make better decisions over time. Memory is automatically loaded into every AI interaction (auto-mode, chat sessions, pipeline steps) with smart selection based on task context.
Problem
Previously, each AI agent session started with a blank slate. Agent 2 had no idea why Agent 1 made certain decisions, leading to:
Solution
Integrated a lightweight, file-based memory system directly into the existing context loader. This means all AI interactions automatically receive relevant project memory without any code changes to individual services. Smart selection ensures only relevant memory files are loaded based on the current task, preventing context bloat.
Architecture Decision: Why File-Based?
Research (including Letta benchmarks) showed that simple filesystem-based memory (74% accuracy) outperformed complex vector+graph systems like Mem0 (68.5%). Every successful AI coding tool (Claude Code, Cursor, Windsurf) uses flat markdown files. This approach:
Memory Structure
Smart Memory Selection
Memory files are scored and selected based on the current task. This prevents loading 40+ memory files and killing context limits.
terminals.mdtags: [authentication, oauth]relevantTo: [login, security]Selection rules:
gotchas.md(common mistakes)Example: If a user asks to "fix the terminal resize bug", the system:
["fix", "terminal", "resize", "bug"]terminals.mdgets 4x boost from category matchgotchas.md)Memory File Format
Each file uses YAML frontmatter for metadata and usage tracking:
How It Works
Loading:
loadContextFiles()takes optionaltaskContext(title + description) to intelligently select relevant memory files. Memory folder is auto-initialized if it doesn't exist.Scoring: Files are scored based on keyword matching against tags, relevantTo, summary, and category name. Usage history acts as a multiplier - files that contributed to successful features rank higher.
Selection: Top 5 most relevant files are loaded (always includes
gotchas.mdand high-importance files). This keeps context usage under control.Recording: After successful feature completion, Claude Haiku extracts learnings from the agent output and categorizes them into appropriate memory files.
Usage Tracking: Files track how often they're loaded, referenced, and contribute to successful features. Files that get loaded but never referenced are demoted over time.
Decision Rationale: The format captures not just WHAT was decided, but WHY, what was REJECTED, and what would BREAK if changed. This prevents future agents from unknowingly reverting intentional decisions.
Changes
New Files
libs/utils/src/memory-loader.ts(~350 lines)initializeMemoryFolder()- Creates memory structure with starter filesparseFrontmatter()/serializeFrontmatter()- YAML metadata handlingappendLearning()- Add learnings to category filesrecordMemoryUsage()- Track which files are referenced in agent outputextractTerms()- Extract keywords from task context for matchingcalculateUsageScore()- Score files based on usage historyincrementUsageStat()- Update usage statisticsformatLearning()- Format learnings for storageModified Files
libs/utils/src/context-loader.tstaskContextoption for smart selection (title + description)maxMemoryFilesoption (default: 5)memoryFilesarray in resultlibs/utils/src/index.tsTaskContext,MemoryFileInfo,MemoryMetadata,UsageStats,LearningEntryinitializeMemoryFolder,appendLearning,recordMemoryUsage,extractTerms,calculateUsageScore, etc.apps/server/src/services/auto-mode-service.tstaskContextfor smart selectionrecordLearningsFromFeature()methodapps/server/src/services/agent-service.tstaskContextfor chat sessionsWhat's NOT Included
Based on research, these were intentionally excluded:
Testing
npm run build:server)Future Improvements
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.