Multimodal Memory System

A brain-inspired memory system for storing, searching, and managing memories across text, images, audio, and video. Features biologically-inspired memory decay, reinforcement, and cross-modal semantic search.

Features

• Multimodal Storage: Store text, images, audio, and video in a unified embedding space
• Cross-Modal Search: Query with any modality, retrieve from any/all modalities (e.g., search images with text)
• Brain-Inspired Memory Decay: Ebbinghaus forgetting curve with importance modulation
• Memory Reinforcement: Strengthen memories through access and explicit reinforcement
• Memory Consolidation: Automatic short-term to long-term transfer based on access patterns
• Associative Retrieval: Graph-based spreading activation for related memories
• REST API: Full-featured API with interactive Swagger documentation
• Multi-Tenant: API key authentication with complete tenant isolation
• Self-Hostable: Run entirely on your own infrastructure with Docker

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        REST API (FastAPI)                        │
│                    Interactive Docs: /docs                       │
├─────────────────────────────────────────────────────────────────┤
│  Encoder Service (GPU)    │  Memory Services    │   Workers     │
│  - CLIP ViT-H-14 (images) │  - Store/Retrieve   │  - Decay      │
│  - Whisper large-v3       │  - Search           │  - Consolidate│
│    (audio/video)          │  - Associate        │  - Cleanup    │
├─────────────────────────────────────────────────────────────────┤
│  PostgreSQL + AGE  │  Qdrant        │  Redis    │  MinIO        │
│  (metadata + graph)│  (vectors)     │  (cache)  │  (files)      │
└─────────────────────────────────────────────────────────────────┘

Quick Start

Prerequisites

• Docker & Docker Compose v2
• NVIDIA GPU with 8GB+ VRAM (recommended)
• NVIDIA Container Toolkit (for GPU support)

1. Clone the Repository

git clone https://github.com/khadimhussain0/memory.git
cd memory

2. Configure Environment

cp .env.example .env
# Edit .env if needed (defaults work for local development)

3. Start All Services

# Using docker compose directly
docker compose up -d

# Or using make (recommended)
make up

This starts:

• api (port 8000) - REST API with Swagger UI
• encoder (port 8001) - ML model service
• worker - Background task processor
• scheduler - Periodic maintenance tasks
• postgres (port 5432) - Metadata & graph database
• qdrant (port 6333) - Vector database
• redis (port 6379) - Cache & task queue
• minio (port 9000/9001) - Object storage

4. Access the Application

• API Documentation (Swagger): http://localhost:8000/docs
• MinIO Console: http://localhost:9001 (login: minioadmin / minioadmin)

Without GPU (CPU Mode)

The encoder service automatically falls back to CPU if no GPU is available. This is slower but works for development and testing.

API Usage

All API endpoints require the X-API-Key header. Default development key: dev-api-key-12345

Quick Example:

# Store a text memory
curl -X POST http://localhost:8000/memories/ \
  -H "X-API-Key: dev-api-key-12345" \
  -F "modality=text" \
  -F "text=Meeting notes: Discussed Q4 roadmap" \
  -F "title=Q4 Planning"

# Search memories
curl -X POST http://localhost:8000/search/ \
  -H "X-API-Key: dev-api-key-12345" \
  -F "text=budget planning"

For complete API documentation with all endpoints, parameters, and examples, see docs/API.md.

Configuration

Copy .env.example to .env and customize as needed:

Setting	Default	Description
DEBUG	true	Enable debug mode
DATABASE_URL	postgresql://...	PostgreSQL connection string
REDIS_URL	redis://...	Redis connection string
QDRANT_URL	http://...	Qdrant vector DB URL
SHORT_TERM_TTL_HOURS	24	How long memories stay in short-term buffer
CONSOLIDATION_THRESHOLD	3	Access count before auto-consolidation
MIN_MEMORY_STRENGTH	0.05	Memories below this are archived
DECAY_CYCLE_HOURS	24	How often decay runs
MAX_UPLOAD_SIZE_MB	100	Maximum file upload size

Development

Quick Commands (Makefile)

make help       # Show all available commands
make up         # Start all services
make down       # Stop all services
make logs       # View logs
make test       # Run tests
make lint       # Run linter
make shell-api  # Open shell in API container
make shell-db   # Open PostgreSQL shell
make clean      # Remove all containers and volumes

Running Tests

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run with coverage
pytest --cov=api --cov=services

Code Formatting

# Check and fix linting issues
ruff check --fix .

# Format code
ruff format .

Local Development (without Docker)

1. Start infrastructure services:

docker compose up -d postgres redis qdrant minio

2. Install dependencies:

pip install -e ".[dev]"

3. Run API:

uvicorn api.main:app --reload --port 8000

4. Run encoder (requires GPU):

uvicorn models.encoder_service:app --port 8001

5. Run worker:

celery -A workers.celery_app worker --loglevel=info

Memory System Concepts

Memory Types

• Episodic: Personal experiences and events
• Semantic: Facts and general knowledge
• Procedural: How-to knowledge and skills

Memory Stages

• Short-term: Recently added, not yet consolidated
• Long-term: Consolidated memories with proven importance
• Archived: Decayed memories kept for potential retrieval

Memory Decay

Memories naturally decay over time following the Ebbinghaus forgetting curve. The decay rate is modulated by:

• Importance score: More important memories decay slower
• Access frequency: Frequently accessed memories are reinforced
• Emotional valence: Emotionally significant memories persist longer

Memory Reinforcement

Memories can be strengthened through:

• Access: Each retrieval reinforces the memory
• Explicit reinforcement: API call to boost strength
• Consolidation: Moving to long-term storage increases durability

Hardware Requirements

Minimum (CPU-only)

• 8GB RAM
• 20GB disk space
• Any modern CPU

Recommended (with GPU)

• 16GB RAM
• 50GB disk space
• NVIDIA GPU with 8GB+ VRAM (RTX 3070 or better)

GPU Compatibility

Tested with:

• NVIDIA RTX 3090
• NVIDIA RTX 4090
• NVIDIA RTX 5090

Troubleshooting

GPU not detected

# Check NVIDIA driver
nvidia-smi

# Check Docker GPU access
docker run --rm --gpus all nvidia/cuda:12.1-base-ubuntu22.04 nvidia-smi

Encoder service slow to start

First startup downloads ML models (~5GB). Subsequent starts use cached models.

Out of memory errors

Reduce batch sizes in encoder service or use a GPU with more VRAM.

Database connection errors

# Check if services are running
docker compose ps

# View logs
docker compose logs postgres
docker compose logs api

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• OpenAI CLIP - Image-text embeddings
• OpenAI Whisper - Speech recognition
• Qdrant - Vector database
• Apache AGE - Graph database extension
• FastAPI - Web framework