Document Chunking & Embedding Utilities

[omeraplak]

1. Overview:

Provide built-in utilities and interfaces for processing documents, specifically splitting large texts into smaller, manageable chunks (chunking) and converting text chunks into numerical vector representations (embedding). This is a foundational capability for Retrieval-Augmented Generation (RAG) patterns, enabling agents to efficiently search and retrieve relevant information from large document sets stored in vector databases.

2. Goals:

• Offer various text chunking strategies (e.g., fixed size, recursive character splitting, semantic chunking).
• Provide flexible configuration options for chunking (e.g., chunk size, overlap).
• Implement interfaces or wrappers for popular embedding models (e.g., OpenAI Ada, Sentence Transformers, local models).
• Ensure efficient handling of document processing and embedding generation.
• Facilitate the integration of chunked and embedded data with vector stores and retriever components.
• Offer clear APIs for developers to use chunking and embedding functions programmatically.

3. Proposed Architecture & Components:

• TextSplitter Interface/Base Class: Defines the core splitText method. Concrete implementations could include:
  • CharacterTextSplitter: Splits based on character count.
  • RecursiveCharacterTextSplitter: Recursive splitting based on separators.
  • (Future) SemanticChunker: Splits based on semantic meaning.
• EmbeddingModel Interface/Base Class: Defines methods like embedDocuments and embedQuery. Concrete implementations would wrap specific embedding providers/models.
• DocumentProcessor: A utility class or set of functions that orchestrate the loading, chunking, and embedding of documents.
• Configuration: Ways to specify chunking strategy, parameters, and the embedding model to use.
• (Optional) VectorStoreManager Integration: Adapters or helpers to easily push embedded chunks into supported vector stores (though the core vector store might be a separate feature).

4. Affected Core Modules:

• Retriever (BaseRetriever): Retrievers will likely consume or interact with embedded data. This feature provides the means to create that data.
• Utils: Core chunking and embedding logic might reside here or in a dedicated new package (e.g., packages/documents).
• Potentially MemoryManager if supporting document ingestion into memory.

5. Acceptance Criteria (Initial MVP):

• Implement a basic RecursiveCharacterTextSplitter.
• Implement an EmbeddingModel wrapper for a common provider (e.g., OpenAI text-embedding-ada-002).
• Provide a simple utility function that takes a document text, splits it using the implemented splitter, and generates embeddings using the implemented model wrapper.
• The function returns structured data (e.g., an array of objects containing chunk text and its embedding vector).
• Basic documentation explains how to use the text splitter and embedding function.

6. Potential Challenges & Considerations:

• Choosing optimal chunking strategies and parameters for different types of documents and downstream tasks.
• Managing dependencies for various embedding models (local vs. API-based).
• Handling rate limits and costs associated with embedding APIs.
• Performance of chunking and embedding large datasets.
• Ensuring compatibility with different vector database schemas and APIs.
• Providing good defaults while maintaining flexibility.

[Dhruv1969Karnwal]

Hi @omeraplak, I'd like to work on this issue. If no one is currently assigned, could you please assign it to me?

[omeraplak]

Hey, Of course @Dhruv1969Karnwal. feel free to take it on! Just a heads-up, it might involve some technical challenges.
Do you have an idea of how you’re planning to implement it? Would love to hear your approach.

[UmeshpJadhav]

Hi @omeraplak 👋, here’s my implementation plan:

I’ll introduce a new packages/documents module to handle document preprocessing and embeddings. It’ll include:

• A RecursiveCharacterTextSplitter for chunking large documents into smaller segments with overlap for context preservation.
• An OpenAIEmbeddingModel wrapper using text-embedding-ada-002 for creating embeddings.
• A DocumentProcessor utility to tie both together — it’ll take raw text, split it, embed the chunks, and return structured objects ready for vector databases or retrievers.

This will serve as an initial MVP integration with the BaseRetriever, laying groundwork for more advanced features like semantic chunking and vector store support later. Please assign me this issue to implement it. 🚀

[omeraplak]

Hi @omeraplak 👋, here’s my implementation plan:

I’ll introduce a new packages/documents module to handle document preprocessing and embeddings. It’ll include:

• A RecursiveCharacterTextSplitter for chunking large documents into smaller segments with overlap for context preservation.
• An OpenAIEmbeddingModel wrapper using text-embedding-ada-002 for creating embeddings.
• A DocumentProcessor utility to tie both together — it’ll take raw text, split it, embed the chunks, and return structured objects ready for vector databases or retrievers.

This will serve as an initial MVP integration with the BaseRetriever, laying groundwork for more advanced features like semantic chunking and vector store support later. Please assign me this issue to implement it. 🚀

Hey, solid plan! 🔥

[UmeshpJadhav]

Hi @omeraplak 👋

I've completed the implementation of the document processing utilities. I've created PR #893 which contains:

• packages/documents: A new package with RecursiveCharacterTextSplitter, OpenAIEmbeddingModel, and DocumentProcessor.
• Core Integration: DocumentRetriever abstract class for building vector-based retrievers.
• Unit tests and documentation.

The implementation follows the plan we discussed. Looking forward to your review! 🚀