A high-performance, context-aware Semantic Caching Layer designed for RAG pipelines (such as NotebookLM). It supports dual-mode deployment: fully local (FastAPI, Redis VSS, Ollama) and Google Cloud serverless (Cloud Run, Compute Engine, Vertex AI Gemini & Embeddings).
It features a premium, responsive Glassmorphic Developer Dashboard styled to match custom portfolio designs.
- Semantic Similarity Search: Evaluates vector embeddings of incoming queries. Semantically identical or rephrased queries are served instantly from the database.
- Context-Aware Pre-Filtering (Redis-Native Hybrid Search):
Prevents cross-context contamination by restricting vector lookup to specific document boundaries:
@context_hash:{context_hash} => [KNN 1 @embedding $vector AS similarity_score] - Dual AI Engines (Ollama / GCP Vertex AI):
- Local Mode: Uses Ollama with
llama3.2:latest(2.0 GB LLM) andnomic-embed-text(embeddings). - Cloud Mode: Serverless integration using Google Vertex AI's
gemini-1.5-flashandtext-embedding-004(embeddings) for zero idle VM costs.
- Local Mode: Uses Ollama with
The project includes an embedded glassmorphic developer console to monitor and test cache performance in real-time.
- Query Simulator: A playground to run prompt inputs and similarity thresholds.
- Pipeline Flow Visualizer: An animating flow diagram that traces execution paths in real-time with neon connection paths (Green for cache Hits, Purple for Misses).
- Real-Time Metrics: Sessions stats including total queries, hit rate, and cumulative latency saved.
- Live Redis Cache Inspector: An interactive database inspector to view keys, verify context hashes, delete individual items, or invalidate entire context tags.
Access the dashboard at http://127.0.0.1:8000/ after starting the server.
Ensure you have Ollama installed and running. Pull the required models:
ollama pull llama3.2
ollama pull nomic-embed-textVector search requires Redis with the RediSearch module. Run it via Docker:
docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 redis/redis-stack:latestpython -m venv venv
# Windows
.\venv\Scripts\activate
# macOS/Linux
source venv/bin/activate
pip install -r requirements.txtCreate a .env file in the root of the project:
OLLAMA_URL=http://localhost:11434
EMBEDDING_MODEL=nomic-embed-text
LLM_MODEL=llama3.2:latest
SIMILARITY_THRESHOLD=0.90
CACHE_TTL=3600
VECTOR_DIMENSION=768
REDIS_URL=redis://localhost:6379
# GCP Vertex AI settings (optional, defaults to false)
USE_VERTEX_AI=false
GCP_PROJECT_ID=your-gcp-project-id
GCP_REGION=us-central1python -m uvicorn app.main:app --reloadThis architecture is optimized to run serverless in GCP, costing virtually $0/month by utilising GCP's Free Tier and Vertex AI's pay-as-you-go pricing.
- Spin up an
e2-microVM (always free tier under US regionsus-central1,us-east1) running Ubuntu. - Install Docker on the VM and launch Redis Stack with a secure password:
docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 -e REDIS_ARGS="--requirepass YOUR_SECURE_PASSWORD" redis/redis-stack:latest
- Authenticate with Google Cloud CLI and set your project:
gcloud auth login gcloud config set project YOUR_PROJECT_ID - Enable the Vertex AI and Artifact Registry APIs:
gcloud services enable aiplatform.googleapis.com artifactregistry.googleapis.com - Submit the Docker build:
gcloud builds submit --tag gcr.io/YOUR_PROJECT_ID/semantic-cache-api
- Deploy the container to Cloud Run (fully managed serverless):
gcloud run deploy semantic-cache-api \ --image gcr.io/YOUR_PROJECT_ID/semantic-cache-api \ --platform managed \ --allow-unauthenticated \ --set-env-vars="REDIS_URL=redis://:YOUR_SECURE_PASSWORD@VM_EXTERNAL_IP:6379,USE_VERTEX_AI=true,GCP_PROJECT_ID=YOUR_PROJECT_ID,GCP_REGION=us-central1"
python -m pytest