RAG Blueprint

A comprehensive open-source framework for building production-ready Retrieval-Augmented Generation (RAG) systems. This blueprint simplifies the development of RAG applications while providing full control over performance, resource usage, and evaluation capabilities.

Open-source community offers a wide range of RAG-related frameworks focus on the specific area - monitoring, visualization or processing. While building or buying RAG systems has become increasingly accessible, deploying them as production-ready data products remains challenging. Our framework bridges this gap by providing a streamlined development experience with easy configuration and customization options, while maintaining complete oversight of performance and resource usage.

It comes with built-in monitoring and observability tools for better troubleshooting, integrated LLM-based metrics for evaluation, and human feedback collection capabilities. Whether you're building a lightweight knowledge base or an enterprise-grade application, this blueprint offers the flexibility and scalability needed for production deployments.

Figure 1: High-level architecture of the RAG Blueprint framework showing the main components and data flow

🚀 Features

• Multiple Knowledge Base Integration: Seamless extraction from several Data Sources(Confluence, Notion, PDF)
• Wide Models Support: Availability of numerous embedding and language models
• Vector Search: Efficient similarity search using vector stores
• Interactive Chat: User-friendly interface for querying knowledge on Chainlit
• Performance Monitoring: Query and response tracking with Langfuse
• Evaluation: Comprehensive evaluation metrics using RAGAS
• Setup flexibility: Easy and flexible setup process of the pipeline

🛠️ Tech Stack

Core

Python • LlamaIndex • Chainlit • Langfuse • RAGAS

---

Data Sources

Notion • Confluence • PDF files • BundestagMine

---

Embedding Models

VoyageAI • OpenAI • Hugging Face

---

Language Models

LiteLLM - Availability of many LLMs via providers like OpenAI, Google or Anthropic as well as local LLMs

---

Vector Stores

Qdrant • Chroma • PGVector

---

Infrastructure

PostgreSQL • Docker

🚀 Quickstart

Check the detailed Quickstart Setup

🏗️ Architecture

Data Flow

1. Extraction:

   • Fetches content from respective data sources
   • Preprocess retrieved resources and parse it to markdown

2. Embedding:

   • Applies markdown aware splitting
   • Embeds final nodes using the selected embedding model
   • Saves the embeddings in the selected vector store

3. Augmentation

   • Defines retrieval and augmentation pipeline encapusalted in a chat engine
   • Integrates Chainlit for UI interface
   • Integrates Langfuse for observability of generated responses and user queries

4. Evaluation:

   • Uses Chainlit and Langfuse platforms for gathering human feedback
   • Employs Ragas package for evaluating perfomance of current setup

For more info refer to specific readmes of Extraction, Embedding, Augmentation and Evaluation.

Integrations

For user interface the codebase uses Chainlit, which is integrated with Langfuse responsible for observability and tracing of the system. Moreover, integration enables building evaluation datasets based on the user feeback regarding the system answers. Feedback is saved in Langfuse datasets and later used by Evaluation module.

📁 Project Structure

.
├── build/            # Build and deployment scripts
│   └── workstation/  # Build scripts for workstation setup
├── configurations/   # Configuration and secrets files
├── data/             # Data for local testing
├── res/              # Assets
└── src/              # Source code
    ├── augmentation/   # Chainlit, Langfuse, and RAG processing components
    ├── core/           # Base package
    ├── extraction/     # Data sources extraction
    ├── embedding/      # Data embedding
    └── evaluate/       # Evaluation system
├── tests/            # Unit tests

📚 Documentation

For detailed documentation on setup, configuration, and development:

• Documentation Site
• Quickstart Setup