📊 Multi-Modal RAG 2.0

An enterprise-grade Retrieval-Augmented Generation system for PDFs, tables, images, charts, and audio.

“Text + Tables + Charts + Audio — All in One RAG System” 🚀

📌 Overview

Multi-Modal RAG 2.0 is a Retrieval-Augmented Generation system designed to process and retrieve information from multiple data modalities instead of relying only on plain text.

The system supports:

📄 PDF Text
📊 Tables
🖼️ Images and Charts
🎙️ Audio and Video Transcriptions

Each modality is converted into searchable information and indexed in ChromaDB. Users can then ask questions through a unified interface, and the system retrieves relevant context before generating an answer with source information.

🎯 Supported Modalities

Modality	Processing	Output
📄 PDF Text	PyPDF2 / Unstructured	Extracted document text
📊 Tables	Camelot / Tabula	Structured table content
🖼️ Images & Charts	Gemini Vision	Text descriptions and insights
🎙️ Audio / Video	Whisper	Speech transcription

🧠 System Architecture

graph TD
    A[PDF / Image / Audio / Video] --> B[Multi-Modal Ingestion Pipeline]

    B --> C[PDF Processor]
    B --> D[Image Processor]
    B --> E[Audio Processor]

    C --> F[Text + Tables]
    D --> G[Image / Chart Descriptions]
    E --> H[Audio Transcriptions]

    F --> I[Embedding Pipeline]
    G --> I
    H --> I

    I --> J[ChromaDB Vector Database]

    K[User Query] --> L[Retriever]
    J --> L

    L --> M[Relevant Multi-Modal Context]
    M --> N[LLM Generator]
    N --> O[Answer + Sources]

    P[Streamlit UI] --> K
    P --> Q[FastAPI Backend]
    Q --> L

🔄 RAG Workflow

Upload File
     │
     ▼
Identify File Type
     │
     ├── PDF ──────► Extract Text + Tables
     │
     ├── Image ────► Generate Description
     │
     └── Audio ────► Generate Transcription
                         │
                         ▼
                    Create Embeddings
                         │
                         ▼
                     Store in ChromaDB
                         │
                         ▼
User Question ───────► Retrieve Relevant Context
                         │
                         ▼
                      LLM Generation
                         │
                         ▼
                  Final Answer + Sources

✨ Key Features

Feature	Description
📄 PDF Text Extraction	Extracts searchable text from PDF documents
📊 Table Extraction	Extracts table content using Camelot or Tabula
🖼️ Image Understanding	Uses Gemini Vision to understand images, charts, and diagrams
🎙️ Audio Transcription	Converts audio and video speech into searchable text
🗃️ Vector Database	Stores embeddings and metadata in ChromaDB
🔍 Semantic Retrieval	Retrieves relevant information based on query meaning
🤖 LLM Generation	Generates answers using Groq or OpenAI-compatible models
🔗 Source References	Returns relevant source information with answers
⚡ FastAPI Backend	Provides REST APIs for ingestion and querying
🖥️ Streamlit Dashboard	Provides an interactive upload and chat interface

🛠️ Tech Stack

Layer	Technology	Purpose
Language	Python 3.11+	Core development
Backend	FastAPI	REST API
Frontend	Streamlit	Interactive dashboard
Orchestration	LangChain	RAG and LLM integration
Vector Database	ChromaDB	Embedding storage and retrieval
Embeddings	Sentence Transformers	Semantic text embeddings
PDF Processing	PyPDF2 / Unstructured	Text extraction
Table Extraction	Camelot / Tabula	PDF table extraction
Vision	Google Gemini	Image and chart understanding
Audio Processing	Whisper	Audio transcription
LLM	Groq / OpenAI	Answer generation

📂 Project Structure

multi-modal-rag/
│
├── src/
│   ├── ingestion/
│   │   ├── pdf_processor.py
│   │   ├── image_processor.py
│   │   ├── audio_processor.py
│   │   ├── vector_index.py
│   │   └── pipeline.py
│   │
│   ├── retrieval/
│   │   └── hybrid_retriever.py
│   │
│   ├── generation/
│   │   └── generator.py
│   │
│   └── app.py
│
├── data/
│   └── raw/
│
├── chroma_db/
│
├── dashboard.py
├── requirements.txt
├── .env
├── .gitignore
└── README.md

🚀 Getting Started

Prerequisites

Before running the project, make sure you have:

Python 3.11+
Conda or virtual environment support
Groq API Key
Google Gemini API Key
OpenAI API Key (optional)

1️⃣ Clone the Repository

git clone https://github.com/vaibhav07772/multi-modal-rag.git
cd multi-modal-rag

2️⃣ Create a Conda Environment

conda create -n multi-rag python=3.11 -y
conda activate multi-rag

3️⃣ Install Dependencies

pip install -r requirements.txt

4️⃣ Configure Environment Variables

Create a .env file in the project root:

GROQ_API_KEY=gsk_xxxxxxxxxxxxxxxxxxxxxx
GOOGLE_API_KEY=your_google_api_key
OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxx

OPENAI_API_KEY is optional if you are using only Groq and Gemini.

5️⃣ Start the FastAPI Backend

uvicorn src.app:app --host 127.0.0.1 --port 8000 --reload

The API server will run at:

http://127.0.0.1:8000

6️⃣ Start the Streamlit Dashboard

Open another terminal, activate the environment, and run:

conda activate multi-rag
streamlit run dashboard.py

The Streamlit application will open at:

http://localhost:8501

🌐 Application URLs

Resource	URL
🖥️ Streamlit Dashboard	`http://localhost:8501`
📡 FastAPI Server	`http://127.0.0.1:8000`
📚 API Documentation	`http://127.0.0.1:8000/docs`
❤️ Health Check	`http://127.0.0.1:8000/health`

📡 API Endpoints

Method	Endpoint	Description
`POST`	`/ingest`	Upload and process PDF, image, or audio files
`POST`	`/ask`	Ask questions using multi-modal context
`GET`	`/health`	Check application health
`GET`	`/docs`	Interactive Swagger API documentation

📤 Example: Ingest a File

POST /ingest
Content-Type: multipart/form-data

file: <binary file>

The system detects the file type and routes it to the appropriate processor.

❓ Example: Ask a Question

Request

{
  "query": "What is the main topic of the document?",
  "top_k": 5
}

Response

{
  "answer": "The document discusses...",
  "sources": [
    {
      "source": "file.pdf",
      "type": "pdf_text"
    }
  ]
}

💡 Example Use Cases

📄 Document Intelligence

Upload technical PDFs and ask questions about their content.

"What are the main requirements mentioned in this document?"

📊 Table Analysis

Extract information from PDF tables.

"What was the highest revenue in the financial table?"

📈 Chart Understanding

Analyze charts and graphs.

"What trend does this chart show?"

🎙️ Meeting or Lecture Analysis

Upload an audio recording and ask:

"What are the main decisions discussed in this meeting?"

🧠 Multi-Source Question Answering

Upload multiple documents and media files and ask questions across all of them.

📊 Multi-Modal Processing

graph LR
    A[PDF] --> E[Knowledge Base]
    B[Tables] --> E
    C[Images / Charts] --> E
    D[Audio / Video] --> E

    E --> F[ChromaDB]
    F --> G[Semantic Retrieval]
    G --> H[LLM]
    H --> I[Grounded Answer]

🖼️ Demo Screenshot

Streamlit Dashboard

Multi-Modal RAG 2.0 Dashboard — Upload PDFs, images, charts, or audio files, process them through the multi-modal RAG pipeline, and ask questions using AI.

🔮 Future Improvements

🤝 Contributing

Contributions are welcome!

Fork the repository
Create a new branch

git checkout -b feature/your-feature-name

Make your changes
Commit your changes

git commit -m "Add new feature"

Push the branch

git push origin feature/your-feature-name

Open a Pull Request

Code Style

Use black for Python formatting
Use isort for import sorting
Add docstrings to important functions
Keep modules clean and modular

📜 License

This project is released under the MIT License.

You are free to use, modify, and distribute this project.

📬 Connect With Me

Vaibhav Singh

GitHub: https://github.com/vaibhav07772
LinkedIn: https://www.linkedin.com/in/vaibhav07772/
Email: vs9502778@gmail.com

⭐ Show Your Support

If you find this project useful, please consider giving the repository a star ⭐.

Your support helps the project reach more developers and AI enthusiasts!

📊 Multi-Modal RAG 2.0

"Text + Tables + Charts + Audio — All in One RAG System" 🚀

Built with ❤️ using Python, LangChain, ChromaDB, FastAPI, Streamlit, Gemini, Whisper, and Groq.

Name		Name	Last commit message	Last commit date
images		images
src		src
.gitignore		.gitignore
README.md		README.md
dashboard.py		dashboard.py
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation