Multimodal RAG System

A complete Multimodal Retrieval-Augmented Generation (RAG) system that can understand PDF documents, images, tables, charts, and diagrams using Docling, ChromaDB, CLIP, LangChain, and Google Gemini.

The system extracts multimodal information from PDF documents, converts it into vector embeddings, stores them in ChromaDB, retrieves the most relevant content, and generates accurate answers using Google Gemini.

---

Features

• PDF Document Processing
• Text Extraction
• Image Extraction
• Image Caption Generation (Gemini Vision)
• CLIP Image Embeddings
• Table Extraction
• Chart Detection
• Semantic Text Chunking
• Vector Search using ChromaDB
• Google Gemini Answer Generation
• Streamlit User Interface
• Retrieval-Augmented Generation (RAG)

---

Architecture

                PDF Document
                     │
                     ▼
             Document Loader
                     │
                     ▼
             Document Parser
                     │
     ┌───────────────┼───────────────┐
     ▼               ▼               ▼
   Text           Images          Tables
     │               │               │
     ▼               ▼               ▼
 Chunking      Image Caption     Table Extraction
     │               │
     ▼               ▼
 Text Embeddings   CLIP Embeddings
      \             /
       \           /
        ▼         ▼
        Chroma Vector Database
                 │
                 ▼
             Retriever
                 │
                 ▼
          Prompt Construction
                 │
                 ▼
           Google Gemini
                 │
                 ▼
           Final Response

---

Project Structure

Multimodal-RAG/
│
├── app.py
├── streamlit_app.py
├── rag_pipeline.py
│
├── config.py
├── logger.py
├── prompts.py
├── data_models.py
│
├── document_loader.py
├── document_parser.py
├── image_extractor.py
├── image_captioner.py
├── image_embeddings.py
├── table_extractor.py
├── chart_extractor.py
├── chunking.py
├── vector_store.py
├── retriever.py
├── llm.py
│
├── requirements.txt
├── README.md
├── .env.example
├── .gitignore
└── rag_pdf_pipeline.png

---

Technologies Used

Programming

• Python

Frameworks

• LangChain
• Streamlit

LLM

• Google Gemini 2.5 Flash

Vector Database

• ChromaDB

Document Processing

• Docling
• PyMuPDF

Embedding Models

• BAAI/bge-m3
• OpenAI CLIP

OCR

• EasyOCR

Deep Learning

• PyTorch
• Transformers

---

Installation

Clone Repository

git clone https://github.com/your-username/Multimodal-RAG-System.git

cd Multimodal-RAG-System

---

Create Virtual Environment

Windows

python -m venv .venv

.venv\Scripts\activate

Linux / macOS

python3 -m venv .venv

source .venv/bin/activate

---

Install Dependencies

pip install -r requirements.txt

---

Environment Variables

Create a .env file in the project root.

GOOGLE_API_KEY=YOUR_GOOGLE_API_KEY

LLM_MODEL=gemini-2.5-flash

EMBEDDING_MODEL=BAAI/bge-m3

CHROMA_COLLECTION_NAME=multimodal_rag

CHUNK_SIZE=1000

CHUNK_OVERLAP=200

TOP_K=5

Replace YOUR_GOOGLE_API_KEY with your own Google Gemini API key.

---

Run CLI Application

python app.py

---

Run Streamlit Application

streamlit run streamlit_app.py

---

Supported Inputs

• PDF Documents
• Images
• Tables
• Charts
• Diagrams

---

Workflow

1. Load PDF document
2. Parse document
3. Extract text
4. Extract images
5. Generate image captions
6. Generate CLIP image embeddings
7. Extract tables
8. Detect charts
9. Perform semantic chunking
10. Store vectors in ChromaDB
11. Retrieve relevant chunks
12. Generate answers using Google Gemini

---

Example Questions

Summarize this document.

Explain the introduction.

Describe the first image.

Explain the table on page 5.

What trend is shown in the chart?

What are the key findings?

Give a concise summary.

Which page discusses drone regulations?

Compare the information in the table and image.

What conclusions are presented?

---

Example Output

Question

Summarize this document.

Answer

This document provides an overview of drone technology,
its applications, market trends, regulations, and future
scope. It also contains tables summarizing specifications
and images illustrating different drone categories.

---

Future Improvements

• Hybrid Search (BM25 + Vector Search)
• Multi-document RAG
• Citation Support
• Conversation Memory
• Agentic RAG
• Knowledge Graph Integration
• FAISS Support
• Milvus Support
• Qdrant Support

---

Screenshots

Architecture

rag_pdf_pipeline.png

Streamlit Interface

---

Author

Venky Eelaprolu

AI/ML Engineer | Machine Learning | Deep Learning | Generative AI

GitHub

https://github.com/Venky-eelaprolu

LinkedIn

https://www.linkedin.com/in/venkyeelaprolu6303266966/

---

Support

If you found this project useful, please consider giving it a ⭐ on GitHub.