Rust Self-Learning Memory

A zero-trust episodic memory backend for AI agents, written in Rust. This system maintains a durable, verifiable record of agent execution while extracting and learning from patterns to improve future decision-making.

Overview

This project provides a production-grade memory system designed for AI agents that need to:

• Record episodes: Start → Execute → Score → Learn → Retrieve lifecycle
• Store durably: Leverage Turso/libSQL for distributed SQL persistence
• Cache efficiently: Use redb for hot-path key-value access
• Extract patterns: Learn decision points, tool sequences, and recovery heuristics
• Retrieve intelligently: Semantic and context-based pattern retrieval
• Verify security: Zero-trust validation with comprehensive security checks

Table of Contents

• Overview
• Architecture
• Features
• Prerequisites
• Installation
• Configuration
• Quick Start
• Command Line Interface
• API Documentation
• Project Structure
• Code Coverage
• Storage
• Performance
• Security
• Development
  • Code Style
  • CI Pipeline
  • Testing
  • Pre-commit Hooks
• Dependencies
• Contributing
• License
• Resources
• Contact

Architecture

┌─────────────────────────────────────────────────────────┐
│                  SelfLearningMemory                     │
│            (Main orchestration interface)               │
└───────────┬─────────────────────────────┬───────────────┘
            │                             │
    ┌───────▼────────┐          ┌─────────▼─────────┐
    │  memory-core   │          │ Episode Lifecycle │
    │                │          │                   │
    │  • Episodes    │          │ • Start           │
    │  • Patterns    │          │ • Log Steps       │
    │  • Heuristics  │          │ • Complete        │
    │  • Learning    │          │ • Score & Learn   │
    └───────┬────────┘          └───────────────────┘
            │
    ┌───────┴─────────────────────────────┐
    │                                     │
    │         Storage Layer               │
    │                                     │
┌───▼──────────────────┐    ┌────────────▼───┐
│  Turso/libSQL        │    │  redb (cache)  │
│  (durable)           │    │  (hot path)    │
│                      │    │                │
│ • episodes table     │    │ • episodes     │
│ • patterns table     │    │ • patterns     │
│ • heuristics table   │    │ • embeddings   │
│ • SQL persistence    │    │ • metadata     │
└──────────────────────┘    └────────────────┘

Features

• Episode Management: Create, log execution steps, complete with scoring
• Pattern Extraction: Automatic extraction of ToolSequences, DecisionPoints, ErrorRecovery
• Learning Queue: Asynchronous pattern learning with backpressure handling
• Dual Storage: Durable Turso/libSQL + fast redb cache
• Security: Zero-trust validation, sanitization, parameterized queries
• MCP Support: Model Context Protocol integration for Claude integration
• Comprehensive Testing: >80% code coverage (currently 84.70%) with unit and integration tests
• Monitoring: Tracing support with structured logging

Prerequisites

Before you begin, ensure you have the following installed:

• Rust: Version 1.70 or higher

  rustup --version  # Verify installation
  rustup override set stable

• Cargo: Comes with Rust (verify with cargo --version)

• System Requirements:

  • Linux, macOS, or Windows
  • Minimum 4GB RAM (8GB recommended)
  • 500MB disk space for dependencies

• Optional:

  • Turso Account: Required for production deployments with durable storage
    • Sign up at turso.tech
    • Create a database and obtain credentials
  • libSQL CLI: For local Turso/libSQL database testing

    # Install libSQL CLI
    brew install tursodatabase/tap/turso  # macOS
    # or download from https://github.com/tursodatabase/libsql

Installation

As a Library

Add this crate to your Cargo.toml:

[dependencies]
memory-core = "0.1"
memory-storage-turso = "0.1"
memory-storage-redb = "0.1"

Or use cargo add:

cargo add memory-core memory-storage-turso memory-storage-redb

Configuration

Environment Variables

The memory system supports both local development and cloud production deployments with Turso/libSQL:

# Production Turso (Cloud) - VERIFIED ✅
export TURSO_DATABASE_URL="libsql://your-database.turso.io"
export TURSO_AUTH_TOKEN="your-auth-token-here"

# Local Development (SQLite) - VERIFIED ✅
export TURSO_DATABASE_URL="file:./data/memory.db"
# export TURSO_AUTH_TOKEN=""  # Empty for local files

# Optional: redb cache configuration
export REDB_CACHE_PATH="./data/cache.redb"
export REDB_MAX_CACHE_SIZE="1000"  # Maximum episodes to cache

✅ Turso Setup Verified: Both local and cloud configurations are fully tested and working. Complete setup is available via config files for both MCP and CLI.

Example .env File

Create a .env file in your project root:

# Turso Configuration (Production)
TURSO_DATABASE_URL=libsql://my-memory-db.turso.io
TURSO_AUTH_TOKEN=eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9...

# Local Development (Alternative to Turso)
# LIBSQL_DATABASE_PATH=./data/memory.db

# redb Cache
REDB_CACHE_PATH=./data/cache.redb
REDB_MAX_CACHE_SIZE=1000

# Logging
RUST_LOG=info,memory_core=debug

Important: Never commit .env files to version control. Add .env to your .gitignore.

Configuration Options

When initializing SelfLearningMemory, you can configure:

use memory_core::{SelfLearningMemory, MemoryConfig};

let config = MemoryConfig {
    turso_url: std::env::var("TURSO_DATABASE_URL")?,
    turso_token: std::env::var("TURSO_AUTH_TOKEN")?,
    redb_path: std::env::var("REDB_CACHE_PATH")
        .unwrap_or_else(|_| "./data/cache.redb".to_string()),
    max_cache_size: std::env::var("REDB_MAX_CACHE_SIZE")
        .unwrap_or_else(|_| "1000".to_string())
        .parse()?,
    ..Default::default()
};

let memory = SelfLearningMemory::new(config).await?;

Quick Start

Basic Usage Example

use memory_core::SelfLearningMemory;
use memory_core::{TaskContext, ExecutionStep, TaskOutcome};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize memory system
    let memory = SelfLearningMemory::new(Default::default()).await?;

    // Start an episode
    let context = TaskContext {
        language: "rust".to_string(),
        domain: "web".to_string(),
        tags: vec!["api".to_string()],
    };

    let episode = memory.start_episode(
        "Build REST API endpoint".to_string(),
        context.clone(),
    ).await?;

    // Log execution steps
    let step = ExecutionStep {
        tool: "rustc".to_string(),
        action: "compile".to_string(),
        latency_ms: 1250,
        tokens: 2500,
        success: true,
        observation: "Compiled successfully".to_string(),
    };

    memory.log_step(episode.id.clone(), step).await?;

    // Complete and score episode
    let outcome = TaskOutcome {
        success: true,
        result: Some("Endpoint created".to_string()),
        error: None,
        duration_ms: 5000,
    };

    let completed = memory.complete_episode(episode.id, outcome).await?;

    // Retrieve similar past episodes
    let relevant = memory.retrieve_relevant_context(
        "Build REST endpoint".to_string(),
        context,
        5,
    ).await?;

    println!("Found {} relevant episodes", relevant.len());
    Ok(())
}

Command Line Interface

The Memory CLI provides a comprehensive command-line interface for managing the Self-Learning Memory System. It offers direct access to all memory operations without writing code.

Installation

# Install with full features (recommended)
cargo install --path memory-cli --features full

# Or install specific features
cargo install --path memory-cli --features turso,redb

Quick CLI Examples

# Configure database connection (Cloud)
echo '[database]
turso_url = "libsql://your-db.turso.io"
turso_token = "your-auth-token"' > memory-cli.toml

# OR for local development
echo '[database]
turso_url = "file:./data/memory.db"
turso_token = ""' > memory-cli.toml

# Create and manage episodes
memory-cli episode create --task "Implement user authentication"
memory-cli episode list --limit 5
memory-cli episode view <episode-id>

# Log execution steps
memory-cli episode log-step <episode-id> \
  --tool "grep" \
  --action "Search patterns" \
  --success true \
  --latency-ms 150

# Complete episodes
memory-cli episode complete <episode-id> --outcome success

# Analyze patterns
memory-cli pattern list --min-confidence 0.8
memory-cli pattern effectiveness --top 10

# Monitor storage health
memory-cli storage health
memory-cli storage stats

# Search episodes
memory-cli episode search "authentication"

CLI Features

• Episode Management: Create, list, view, complete, and search episodes
• Pattern Analysis: List patterns, analyze effectiveness, manage pattern decay
• Storage Operations: Health checks, statistics, synchronization, vacuum operations
• Multiple Output Formats: Human-readable, JSON, and YAML output
• Shell Completions: Auto-generated completions for Bash, Zsh, and Fish
• Dry Run Mode: Preview operations before execution
• Verbose Logging: Detailed diagnostics and error information

Configuration

The CLI supports multiple configuration formats (TOML, JSON, YAML) and locations. See the Configuration Guide for complete documentation.

Quick Setup:

[database]
turso_url = "libsql://your-db.turso.io"
turso_token = "your-auth-token"
redb_path = "memory.redb"

[storage]
max_episodes_cache = 1000
cache_ttl_seconds = 3600
pool_size = 10

[cli]
default_format = "human"
progress_bars = true
batch_size = 100

Advanced Usage

Scripting Integration

#!/bin/bash
# Batch process recent episodes

memory-cli episode list --limit 10 --format json | \
  jq -r '.episodes[].episode_id' | \
  while read episode_id; do
    memory-cli episode view "$episode_id" --format json > "episode_$episode_id.json"
  done

Health Monitoring

#!/bin/bash
# System health check

if ! memory-cli storage health --format json | jq -e '.overall == "healthy"' > /dev/null; then
  echo "Storage health check failed!" >&2
  exit 1
fi

Complete Documentation

For comprehensive CLI documentation, see CLI User Guide which includes:

• Complete command reference with all options
• Configuration examples and templates
• Scripting examples and best practices
• Troubleshooting guide
• Integration patterns

API Documentation

Comprehensive API documentation is available at:

• docs.rs/memory-core - Core memory system API
• docs.rs/memory-storage-turso - Turso storage backend
• docs.rs/memory-storage-redb - redb cache backend

You can also generate and view the documentation locally:

# Generate and open documentation for all crates
cargo doc --all --open

# Generate documentation with private items
cargo doc --all --document-private-items --open

Project Structure

.
├── memory-core/              # Core memory system
│   ├── src/
│   │   ├── episode/          # Episode management
│   │   ├── pattern/          # Pattern types and operations
│   │   ├── patterns/         # Pattern extraction and learning
│   │   ├── learning/         # Learning queue and orchestration
│   │   ├── memory.rs         # Main SelfLearningMemory struct
│   │   └── lib.rs
│   └── tests/
├── memory-storage-turso/     # Turso/libSQL backend
├── memory-storage-redb/      # redb cache backend
├── memory-mcp/               # MCP protocol support
├── test-utils/               # Shared test utilities
├── .github/workflows/
│   ├── ci.yml               # Main CI pipeline with coverage gate
│   ├── security.yml         # Security scanning
│   └── release.yml          # Release automation
├── .codecov.yml             # Codecov configuration
└── AGENTS.md                # Agent guidelines and workflows

Code Coverage

This project enforces >80% code coverage on the main branch via cargo-llvm-cov in the CI pipeline. Current coverage: 84.70%.

• Coverage reports generated as HTML and LCOV format
• Uploaded to Codecov for tracking and analysis
• Coverage badge above shows current status
• See .codecov.yml for configuration

Running Coverage Locally

# Install cargo-llvm-cov
cargo install cargo-llvm-cov

# Generate coverage report (HTML)
cargo llvm-cov --all-features --workspace --html

# Generate LCOV format (for Codecov)
cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info

# View summary
cargo llvm-cov --all-features --workspace --summary-only

Storage

Turso/libSQL (Durable) ✅ VERIFIED

• Cloud Production: Distributed SQL database with TLS encryption
• Local Development: SQLite-compatible file storage for offline development
• Tables: episodes, patterns, heuristics
• Parameterized queries for security
• Connection pooling and circuit breaker pattern
• Status: Both local and cloud configurations fully tested and working

redb (Cache) ✅ VERIFIED

• Embedded key-value store for hot-path access
• Tables: episodes, patterns, embeddings, metadata
• Synchronous operations (wrapped in spawn_blocking)
• Reconciliation with Turso via sync_memories()
• Bincode serialization with security limits

Performance

This project maintains strict performance baselines and tracks regressions via automated benchmarks. All operations significantly exceed target performance requirements.

Baseline Metrics (P95)

Operation	Actual	Target	Status
Episode Creation	2.56 µs	< 50ms	19,531x faster ✓
Step Logging	1.13 µs	< 20ms	17,699x faster ✓
Episode Completion	3.82 µs	< 500ms	130,890x faster ✓
Pattern Extraction	10.43 µs	< 1000ms	95,880x faster ✓
Memory Retrieval	721 µs	< 100ms	138x faster ✓
Storage (Write)	13.22 ms	< 50ms	3.8x faster ✓

See PERFORMANCE_BASELINES.md for detailed metrics and analysis.

Running Benchmarks

# Run all benchmarks
cargo bench --package memory-benches

# View detailed reports
open target/criterion/report/index.html

# Check for regressions
./scripts/check_performance_regression.sh

CI Performance Tracking

Benchmarks run automatically on:

• Every push to main branch
• All pull requests
• Weekly schedule (Mondays)

Performance regressions >10% trigger automatic alerts. See .github/workflows/benchmarks.yml for configuration.

Security

See SECURITY.md for detailed security guidelines including:

• Zero-trust validation principles
• Credential handling (environment variables only)
• Input sanitization
• Parameterized SQL queries
• RBAC and access control

Development

Development Setup

To set up the project for development and contribution:

# Clone the repository
git clone https://github.com/d-o-hub/rust-self-learning-memory.git
cd rust-self-learning-memory

# Ensure you're using stable Rust
rustup override set stable

# Build all crates
cargo build --all

# Run the test suite
cargo test --all

# Run with debug logging
RUST_LOG=debug cargo test --all -- --nocapture

Code Style

• Rust: Follow rustfmt and Clippy rules
• Keep files ≤ 500 LOC (split into submodules if needed)
• Use anyhow::Result for top-level functions
• Use thiserror for typed errors
• Document all public APIs

CI Pipeline

The project uses GitHub Actions for continuous integration:

Jobs:
  - format:           Verify code formatting (cargo fmt)
  - clippy:           Lint checks (cargo clippy -D warnings)
  - test:             Test suite (multiple OS, stable Rust)
  - build:            Release build with timing
  - coverage:         Code coverage with >80% gate (currently 84.70%)
  - security-audit:   Vulnerability scanning (cargo audit)
  - supply-chain:     License & advisory checks (cargo-deny)

Testing & Benchmarking

Unit & Integration Tests

# Run all tests with debug logging
RUST_LOG=debug cargo test --all -- --nocapture

# Run specific test
cargo test --all memory::tests::test_start_episode

# Run with coverage
cargo llvm-cov --all-features --workspace

Comprehensive Benchmarking

The project includes a comprehensive benchmark suite covering:

• Concurrent Operations: YCSB-like workloads with configurable read/write ratios
• Memory Pressure Testing: Memory usage monitoring under various scenarios
• Scalability Analysis: Performance scaling across dataset sizes, users, and query complexity
• Multi-Backend Comparison: Performance comparison between redb and Turso/libSQL

Running All Benchmarks

# Run the complete benchmark suite (recommended)
./scripts/run_comprehensive_benchmarks.sh

# Or run individual benchmark suites
./scripts/run_comprehensive_benchmarks.sh run concurrent_operations
./scripts/run_comprehensive_benchmarks.sh run memory_pressure
./scripts/run_comprehensive_benchmarks.sh run scalability
./scripts/run_comprehensive_benchmarks.sh run multi_backend_comparison

# List available benchmark suites
./scripts/run_comprehensive_benchmarks.sh list

Running Specific Benchmarks with Criterion

# Run individual benchmark suites directly
cargo bench -p memory-benches --bench concurrent_operations
cargo bench -p memory-benches --bench memory_pressure
cargo bench -p memory-benches --bench scalability
cargo bench -p memory-benches --bench multi_backend_comparison

# Run all benchmarks (may take considerable time)
cargo bench -p memory-benches

Benchmark Results & Analysis

Results are stored in target/criterion/ with:

• HTML Reports: Interactive charts and detailed statistics
• JSON Data: Raw benchmark measurements for analysis
• Comparative Analysis: Historical performance tracking

Key benchmark categories:

1. Concurrent Operations (concurrent_operations.rs)

   • YCSB Workload A-F patterns (Update Heavy, Read Mostly, etc.)
   • Configurable concurrency levels (1, 4, 8, 16 threads)
   • Measures throughput and latency under concurrent load

2. Memory Pressure (memory_pressure.rs)

   • Steady state, burst load, and gradual growth scenarios
   • Real-time memory monitoring with sysinfo
   • Peak memory usage and variance analysis

3. Scalability (scalability.rs)

   • Dataset size scaling (100 to 5000+ episodes)
   • Concurrent user scaling (1 to 50+ users)
   • Query complexity and batch operation analysis

4. Multi-Backend Comparison (multi_backend_comparison.rs)

   • Performance comparison between redb and Turso/libSQL
   • Write performance, read performance, bulk operations
   • Concurrent performance across backends

CI/CD Integration

Benchmarks run automatically on:

• Main branch pushes: Full benchmark suite with result storage
• Pull requests: Performance regression detection
• Weekly schedule: Long-term performance trend analysis

Performance regressions trigger:

• ❌ Alert: >110% performance degradation
• 📊 PR Comments: Benchmark result summaries
• 📈 Historical Tracking: Performance trend visualization

Interpreting Results

Performance Targets (from plans/PERFORMANCE_BASELINES.md):

• Episode Creation: < 50ms (currently ~2.5µs - 19,531x faster)
• Step Logging: < 20ms (currently ~1.1µs - 17,699x faster)
• Episode Completion: < 500ms (currently ~3.8µs - 130,890x faster)
• Pattern Extraction: < 1000ms (currently ~10µs - 98,768x faster)
• Memory Retrieval: < 100ms (currently ~720µs - 138x faster)

Key Metrics to Monitor:

• Throughput: Operations per second under various loads
• Latency P95: 95th percentile response times
• Memory Usage: Peak and average memory consumption
• Scalability Factor: How performance changes with load
• Backend Efficiency: Storage size vs. operation count

Troubleshooting Benchmarks

Common Issues:

1. Timeout Errors: Increase BENCH_TIMEOUT in the runner script
2. Memory Issues: Reduce concurrency or dataset sizes
3. Inconsistent Results: Run benchmarks multiple times for statistical significance
4. Missing Dependencies: Ensure all benchmark dependencies are installed

Debug Mode:

# Run with debug output
RUST_LOG=debug cargo bench -p memory-benches --bench concurrent_operations

# Run single benchmark iteration for debugging
cargo bench -p memory-benches --bench concurrent_operations -- --verbose

Pre-commit Hooks

This project uses Claude Code hooks for validation:

• Code formatting check
• Clippy linting
• Cargo audit (security)
• Cargo deny (licenses & advisories)
• Test execution
• Secret scanning

Dependencies

Key dependencies:

• tokio: Async runtime
• libsql: Turso/libSQL client
• redb: Embedded key-value store
• serde: Serialization/deserialization
• tracing: Structured logging
• anyhow/thiserror: Error handling
• uuid/chrono: Identifiers and timestamps

Contributing

See CONTRIBUTING.md for contribution guidelines.

Contributions are welcome! Please ensure:

• All tests pass: cargo test --all
• Code is formatted: cargo fmt
• No clippy warnings: cargo clippy -- -D warnings
• Coverage maintained: coverage reports generated

License

This project is licensed under the MIT License - see LICENSE file for details.

Resources

• AGENTS.md - Agent responsibilities and task templates
• TESTING.md - Testing infrastructure and best practices
• SECURITY.md - Security guidelines and threat model
• CONTRIBUTING.md - Contribution guidelines
• ROADMAP.md - Project roadmap and future features