Personal Knowledge Assistant

A lightweight Retrieval-Augmented Generation (RAG) system built in Python. This project lets you query your personal or professional documents using natural language. It processes diverse document types, embeds content with metadata, and retrieves the most relevant chunks using a FAISS HNSW vector database for accurate, context-aware responses.

---

🚀 Features

• Multi-format Document Ingestion: Supports PDF (including scanned/OCR), DOCX, TXT, HTML, and JSON.
• Robust Text Cleaning: Removes noisy OCR artifacts, headers/footers, and extracts metadata.
• Adaptive Chunking: Smart chunking strategies based on document type (Q&A vs. regular).
• High-quality Embeddings: Uses sentence-transformers with BAAI/bge-base-en-v1.5 for efficient embeddings.
• Fast Semantic Search: Powered by FAISS HNSW for scalable and accurate retrieval.
• Context-aware Responses: Combines metadata and intent detection for precise answers.
• Easy Extensibility: Modular pipeline for adding new formats, models, or chunking strategies.

---

🏗️ Project Structure

personal-knowledge-assistant/
├── main.py                      # Entry point for the pipeline
├── rag/
│   ├── generation.py            # Response generator module
│   ├── query_processor.py       # Query parsing, intent & metadata detection
│   └── retriever.py             # Handles semantic retrieval from FAISS
├── utils/
│   ├── document_processor.py    # Orchestrates ingestion, cleaning, chunking
│   ├── embedding_utils.py       # Embedding pipeline (sentence-transformers)
│   └── faiss_utils.py           # FAISS index building, saving, and loading
├── ingestion/
│   ├── loader.py                # Multi-format document loader (PDF, DOCX, etc.)
│   └── cleaner.py               # Text cleaning & metadata extraction
├── chunking/
│   ├── chunking_pipeline.py     # Q&A detection and adaptive chunking selection
│   ├── sliding_window.py        # Sliding window chunker for Q&A docs
│   └── recursive_chunker.py     # Recursive chunker for regular docs
├── vector_db/
│   └── faiss.py                 # FAISS HNSW storage and search implementation
├── docs/                        # Put your source documents here
└── requirements.txt             # Project dependencies

---

🧑‍💻 Comprehensive Installation & Run Guide

1. Prerequisites

• Python 3.8+ (recommended: Python 3.10 or newer)
• Git
• pip (Python package manager)
• Tesseract OCR (optional, only needed for OCR on scanned PDFs)

2. Clone the Repository

git clone https://github.com/mdadnanshuvo/personal-knowledge-assistant.git
cd personal-knowledge-assistant

3. Create and Activate a Virtual Environment

Why?
A virtual environment ensures all dependencies are isolated and do not interfere with your system Python or other projects.

On Windows

python -m venv venv
venv\Scripts\activate

On macOS/Linux

python3 -m venv venv
source venv/bin/activate

4. Upgrade pip (Recommended)

pip install --upgrade pip

5. Install Required Python Dependencies

pip install -r requirements.txt

Extra notes:

• If you want OCR support for scanned PDFs, install the following system packages and Python libraries:
  • System:
    • On Ubuntu/Debian: sudo apt-get install tesseract-ocr
    • On Mac: brew install tesseract
  • Python:

    pip install pytesseract pdf2image pillow

6. Configure Environment Variables

Create a .env file in the root directory (if your project supports custom environment variables).

Common usage (if needed):

# .env file example
MODEL_NAME=BAAI/bge-base-en-v1.5
FAISS_INDEX_NAME=my_knowledge_base

Note:
If the code does not auto-load .env files, you can set environment variables in your shell before running.

On Windows

set MODEL_NAME=BAAI/bge-base-en-v1.5

On macOS/Linux

export MODEL_NAME=BAAI/bge-base-en-v1.5

7. Add Your Documents

• Put your PDFs, DOCX, TXT, Markdown, or JSON files in the docs/ directory.
• Each run will process all files in this directory.

8. Run the Project

python main.py

What happens:

• Documents are loaded, cleaned, chunked, embedded, and indexed.
• You can query your knowledge base using natural language.

9. Troubleshooting

• OCR Not Working:
  • Ensure Tesseract OCR is installed and accessible in your PATH.
• CUDA/FP16 Issues:
  • If you encounter GPU/FP16 errors, set use_fp16=False in your model initialization.
• Out of Memory:
  • For large document sets, ensure your machine has enough RAM; consider chunking/tuning parameters.

10. Deactivate the Virtual Environment

When done, deactivate with:

deactivate

11. Updating Dependencies

To update all Python packages (optional):

pip install --upgrade -r requirements.txt

12. Uninstalling / Cleaning Up

To remove the virtual environment and dependencies:

deactivate
rm -rf venv

13. Advanced Usage

• Change Embedding Model:
  Edit utils/embedding_utils.py and change the model name.
• Tune FAISS Parameters:
  Edit utils/faiss_utils.py or vector_db/faiss.py.
• Add More Document Types:
  Extend ingestion/loader.py.

14. Example Workflow

git clone https://github.com/mdadnanshuvo/personal-knowledge-assistant.git
cd personal-knowledge-assistant
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
# (Optional: pip install pytesseract pdf2image pillow)
# (Optional: install tesseract-ocr system package)
# Add your documents to docs/
python main.py

---

Enjoy querying your personal knowledge base!

⚡ Usage

Run the assistant:

python main.py

Sample output:

Processing documents...
Building FAISS index...
Query: What was the deadline of submitting Scrappy assignment?
Best Answer:
[answer generated based on your documents]

---

🧠 How It Works

1. Document Ingestion: Loads and processes all documents in docs/.
2. Cleaning & Metadata: Cleans text, removes noise, extracts metadata like titles/sections.
3. Chunking: Splits content into chunks, adapting to Q&A or regular formats.
4. Embedding: Converts chunks into embedding vectors using BAAI/bge-base-en-v1.5 via sentence-transformers.
5. Indexing: Stores vectors in a FAISS HNSW index for fast retrieval.
6. Querying: Processes user queries, extracts intent, retrieves relevant chunks.
7. Generation: Synthesizes a context-aware answer from retrieved chunks.

---

🤖 Embedding Model

• Model: BAAI/bge-base-en-v1.5
• Library: sentence-transformers, version 5.1.1
• Code Example:

  from sentence_transformers import SentenceTransformer
  model = SentenceTransformer("BAAI/bge-base-en-v1.5")
  embeddings = model.encode(["your text here"])

---

📁 Example Document Types Supported

• Academic PDFs
• Meeting notes (.docx, .txt)
• Markdown files
• Exported chat logs
• Scanned documents (with OCR)

---

📝 Customization

• Add new chunking strategies: Extend chunking/.
• Switch embedding models: Modify embedding_utils.py to use any HuggingFace model.
• Tune FAISS parameters: Adjust in faiss_utils.py and vector_db/faiss.py.

---

📚 References

• FAISS: Facebook AI Similarity Search
• BAAI/bge-base-en-v1.5 Model Card
• Sentence Transformers Documentation

---

💡 License

MIT License. See LICENSE for details.

---

🙋‍♂️ Author

• mdadnanshuvo

---