Skip to content

Feat: Added FaissDB to backend #217

Merged
willingWill17 merged 18 commits intomainfrom
add/faiss
Aug 21, 2025
Merged

Feat: Added FaissDB to backend #217
willingWill17 merged 18 commits intomainfrom
add/faiss

Conversation

@willingWill17
Copy link
Collaborator

@willingWill17 willingWill17 commented Aug 19, 2025

🚀 PR Description: Add FAISS Vector Storage Backend,

📋 Summary

This PR introduces a new FAISS (Facebook AI Similarity Search) backend for the vector storage system. It enables high-performance similarity search with local file persistence, supporting multiple distance metrics and efficient in-memory indexing.

Usage

VECTOR_STORE_TYPE=faiss
FAISS_BASE_STORAGE_PATH=path/to/your/folder

✨ Features

  • Core Backend Implementation

    • FaissBackend class implementing the VectorStore interface
    • Distance metrics supported: Cosine, Euclidean, Inner Product (IP)
    • Local file persistence with JSON-based storage
    • In-memory FAISS index via faiss-node
    • Automatic vector normalization for cosine distance
  • Testing Coverage

    • Unit tests for connect, insert, search, update, delete, and list operations
    • Metric-specific tests for Euclidean and IP scoring
    • Error handling tests for invalid operations and connection issues
    • Filter support tests: range queries, any filters
    • Mock implementations for file system and FAISS interactions
  • Configuration Integration

    • Environment variable support: FAISS_BASE_STORAGE_PATH
    • Schema validation with Zod
    • Factory integration for backend selection
    • Default configuration with sensible defaults

⚙️ Key Implementation Details

  • Distance Metrics

    • Cosine → IndexFlatIP + normalization
    • Euclidean → IndexFlatL2
    • Inner Product → IndexFlatIP
  • Data Persistence

    • JSON storage per collection
    • Atomic file operations with error handling
    • Lazy loading on connection
    • Automatic directory creation for persistence path
  • Search Capabilities

    • Configurable similarity search result limits
    • Distance-based scoring per metric
    • Metadata filters (including range operators: gte, lte, gt, lt)

✅ Testing Highlights

  • Constructor validation for all distance metrics
  • Robust connection lifecycle tests
  • CRUD operation coverage
  • Search with score validation
  • Advanced filter queries
  • File persistence using mocked FS

🔗 Related Issues

@willingWill17 willingWill17 merged commit 827abd5 into main Aug 21, 2025
5 checks passed
@willingWill17 willingWill17 changed the title Added FaissDB to backend Feat: Added FaissDB to backend Aug 21, 2025
@willingWill17 willingWill17 deleted the add/faiss branch August 21, 2025 17:17
Ptah-CT pushed a commit to DerAuctor/ct-cipher that referenced this pull request Oct 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants