LLM Vector Search

A comprehensive Go-based application that provides document processing, semantic search, and intelligent Q&A capabilities using OpenAI API and vector databases.

Features

• 📄 Document Processing: Process various document types (PDF, text, HTML, URLs)
• 🔍 Semantic Search: Advanced vector-based search with similarity scoring
• 🤖 Intelligent Q&A: AI-powered question answering with context retrieval
• 💾 Vector Storage: Support for PostgreSQL with pgvector and Weaviate
• 🚀 REST API: Complete HTTP API for all functionalities
• 🖥️ CLI Interface: Command-line tools for easy interaction
• ⚙️ Configurable: Flexible configuration with YAML and environment variables

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Documents     │    │   OpenAI API    │    │ Vector Database │
│  (PDF, Text,    │───▶│   (Embeddings   │───▶│  (PostgreSQL/   │
│   URLs, etc.)   │    │   & Chat)       │    │   Weaviate)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────────────────────────────────────────────────────┐
│                    LLM Vector Search                            │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌──────────┐  │
│  │ Document    │ │ Semantic    │ │     Q&A     │ │ REST API │  │
│  │ Processing  │ │   Search    │ │   Engine    │ │ & CLI    │  │
│  └─────────────┘ └─────────────┘ └─────────────┘ └──────────┘  │
└─────────────────────────────────────────────────────────────────┘

Quick Start

Prerequisites

• Go 1.21+
• PostgreSQL 12+ with pgvector extension OR Weaviate
• OpenAI API key

Installation

1. Clone the repository:

git clone <repository-url>
cd llm-vector-search

2. Install dependencies:

go mod tidy

3. Set up your configuration:

cp .env.example .env
# Edit .env with your API keys and database settings

4. Configure the application:

cp config.yaml config.local.yaml
# Edit config.local.yaml as needed

Database Setup

Option 1: PostgreSQL with pgvector

-- Install pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create database
CREATE DATABASE vectordb;

Option 2: Weaviate

Start Weaviate using Docker:

docker run -d \
  --name weaviate \
  -p 8080:8080 \
  -e QUERY_DEFAULTS_LIMIT=25 \
  -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
  semitechnologies/weaviate:latest

Usage

CLI Usage

1. Process a document:

# Process a text file
go run cmd/main.go process file document.txt --title "My Document"

# Process a URL
go run cmd/main.go process url https://example.com/article

# Process raw text
go run cmd/main.go process text "Your text content here"

2. Search documents:

go run cmd/main.go search "your search query" --limit 5 --threshold 0.7

3. Ask questions:

go run cmd/main.go qa "What is the main topic discussed in the documents?"

4. Start the API server:

go run cmd/server/main.go

API Usage

Start the server:

go run cmd/server/main.go

Process a document:

curl -X POST http://localhost:8080/api/v1/documents \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Your document content here",
    "title": "Document Title",
    "source_type": "text"
  }'

Search documents:

curl -X POST http://localhost:8080/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "your search query",
    "limit": 10,
    "threshold": 0.7
  }'

Ask a question:

curl -X POST http://localhost:8080/api/v1/qa \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What is the main topic?",
    "max_tokens": 500
  }'

Configuration

Environment Variables

OPENAI_API_KEY=your_openai_api_key_here
DB_PASSWORD=your_database_password_here

Configuration File

The config.yaml file contains all configuration options:

server:
  host: "localhost"
  port: 8080

openai:
  api_key: "${OPENAI_API_KEY}"
  model: "gpt-3.5-turbo"
  embed_model: "text-embedding-ada-002"
  max_tokens: 1000
  temperature: 0.7

vector:
  provider: "postgres"  # or "weaviate"
  host: "localhost"
  port: 5432
  index: "documents"
  dimension: 1536

database:
  host: "localhost"
  port: 5432
  user: "postgres"
  password: "${DB_PASSWORD}"
  dbname: "vectordb"
  sslmode: "disable"

API Endpoints

Document Processing

• POST /api/v1/documents - Process a new document
• GET /api/v1/documents/:id - Get document by ID
• DELETE /api/v1/documents/:id - Delete a document

Search

• GET /api/v1/search - Search documents (query parameters)
• POST /api/v1/search - Search documents (JSON body)
• GET /api/v1/documents/:id/similar - Find similar documents

Q&A

• POST /api/v1/qa - Ask a question
• POST /api/v1/qa/followup - Generate follow-up questions

Utilities

• GET /health - Health check
• POST /api/v1/embeddings - Create embeddings

Examples

Example 1: Process and Search Scientific Papers

# Process a PDF paper
go run cmd/main.go process file paper.pdf --source-type "scientific_paper"

# Search for specific concepts
go run cmd/main.go search "machine learning algorithms" --limit 5

# Ask questions about the content
go run cmd/main.go qa "What are the main contributions of this paper?"

Example 2: Build a Knowledge Base from URLs

# Process multiple URLs
go run cmd/main.go process url https://docs.example.com/guide1
go run cmd/main.go process url https://docs.example.com/guide2

# Search the knowledge base
go run cmd/main.go search "installation instructions"

# Get detailed answers
go run cmd/main.go qa "How do I install the application?"

Development

Project Structure

llm-vector-search/
├── cmd/                    # Command-line applications
│   ├── main.go            # CLI application
│   └── server/            # HTTP server
├── internal/              # Private application code
│   ├── config/           # Configuration management
│   ├── openai/           # OpenAI API client
│   ├── vector/           # Vector database interfaces
│   ├── document/         # Document processing
│   ├── search/           # Search engine
│   └── qa/               # Q&A engine
├── pkg/                   # Public library code
│   ├── models/           # Data models
│   └── utils/            # Utilities
├── api/                   # HTTP API handlers
├── docs/                  # Documentation
└── examples/              # Example usage

Building

# Build CLI
go build -o bin/llm-vector-search cmd/main.go

# Build server
go build -o bin/server cmd/server/main.go

Testing

go test ./...

Supported Document Types

• Text files: .txt, .md
• PDF files: .pdf
• Web pages: HTTP/HTTPS URLs
• Raw text: Direct text input

Vector Databases

PostgreSQL with pgvector

• High performance for structured data
• ACID compliance
• Familiar SQL interface
• Built-in filtering and indexing

Weaviate

• Purpose-built for vector operations
• Advanced filtering capabilities
• Built-in ML models
• GraphQL API

Performance Considerations

• Chunk Size: Optimal chunk size is typically 500-1500 characters
• Overlap: 10-20% overlap between chunks improves context preservation
• Batch Processing: Use batch operations for multiple documents
• Indexing: Ensure proper vector indexing for fast search

Troubleshooting

Common Issues

1. "OpenAI API key is required"

   • Set the OPENAI_API_KEY environment variable
   • Check your API key validity

2. "Failed to connect to database"

   • Verify database connection settings
   • Ensure PostgreSQL is running and accessible
   • Check if pgvector extension is installed

3. "Vector dimension mismatch"

   • Ensure the vector dimension matches your embedding model
   • OpenAI text-embedding-ada-002 uses 1536 dimensions

4. Poor search results

   • Adjust the similarity threshold
   • Check document processing quality
   • Verify embeddings are being created correctly

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• OpenAI for the powerful API
• PostgreSQL and pgvector teams
• Weaviate for vector database capabilities
• The Go community for excellent libraries