feat: Standardize Query DSL and Enhance Adapter Architecture

[thewebscraping]

Standardize Query DSL and Enhance Adapter Architecture

Summary

This PR introduces a comprehensive Query DSL system with universal operator support across all vector database backends, enhances adapter capabilities, and improves the engine's document retrieval logic.

Key Changes

🎯 Query DSL System

• New Q Class: Introduced composable query objects supporting 8 universal operators ($eq, $ne, $gt, $gte, $lt, $lte, $in, $nin)
• Backend Compilers: Implemented compiler system for AstraDB, ChromaDB, Milvus, and PgVector
• Universal Format: Standardized filter format that compiles to backend-specific syntax
• Nested Field Support: Added field__subfield__operator syntax for nested metadata queries

🔧 Adapter Enhancements

PgVector

• Added nested JSONB path support using #>> operator for dot-notation queries
• Implemented numeric casting ::numeric for type-safe comparisons
• Enhanced WHERE clause generation with proper SQL escaping

All Adapters

• Added capability flags: supports_metadata_only, REQUIRES_VECTOR, SUPPORTS_NESTED
• Integrated where compilers for automatic Q/dict to native format conversion
• Improved error messages with structured exceptions
• Added similarity score injection in search results

🚀 Engine Improvements

• Enhanced get_or_create: Multi-step lookup strategy (pk → vector similarity → metadata)
• Enhanced update_or_create: Improved logic with separate create_defaults support
• Better Embedding Management: Automatic re-embedding on text updates
• Type Safety: Added proper type hints and validation

⚙️ Configuration

• Simplified settings keys (e.g., CHROMA_HOST instead of CHROMA_HTTP_HOST)
• Added VECTOR_DIM setting for default embedding dimension
• Standardized all adapter environment variable names

🧪 Testing

• Added comprehensive Query DSL test suite (test_querydsl.py, test_compilers.py, test_adapters_where.py)
• Updated engine tests for new get_or_create/update_or_create behavior
• Improved mock adapters for better test coverage

📚 Documentation

• Added docs/querydsl.md - Complete Query DSL guide
• Added docs/architecture.md - System design and patterns
• Updated docs/adapters/databases.md - Backend-specific capabilities
• Updated all docs to reflect new API and features

🗑️ Cleanup

• Removed deprecated scripts/tests/ directory (old integration test scripts)
• Removed legacy querydsl/filters/ system
• Consolidated utility functions

Breaking Changes

Configuration Keys

• CHROMA_HTTP_HOST → CHROMA_HOST
• CHROMA_HTTP_PORT → CHROMA_PORT
• CHROMA_CLOUD_TENANT → CHROMA_TENANT
• CHROMA_CLOUD_DATABASE → CHROMA_DATABASE
• MILVUS_USER + MILVUS_PASSWORD → MILVUS_API_KEY

API Changes

• where parameter now accepts Q objects or universal dict format
• Search results now include score in metadata when available
• Adapter class names: ChromaDBAdapter → ChromaAdapter, MilvusDBAdapter → MilvusAdapter, PGVectorAdapter → PgVectorAdapter

Migration Guide

Updating Configuration

# Old
os.environ["CHROMA_HTTP_HOST"] = "localhost"
os.environ["MILVUS_USER"] = "user"
os.environ["MILVUS_PASSWORD"] = "pass"

# New
os.environ["CHROMA_HOST"] = "localhost"
os.environ["MILVUS_API_KEY"] = "token"  # or leave empty for local

Using Query DSL

# Old - dict with implicit $eq
results = engine.search(where={"category": "tech", "year": 2024})

# New - Q objects with explicit operators
from crossvector.querydsl import Q

results = engine.search(where=Q(category="tech") & Q(year__gte=2024))
# Or still use universal dict format
results = engine.search(where={"category": {"$eq": "tech"}, "year": {"$gte": 2024}})

Adapter Imports

# Old
from crossvector.dbs.chroma import ChromaDBAdapter
from crossvector.dbs.milvus import MilvusDBAdapter
from crossvector.dbs.pgvector import PGVectorAdapter

# New
from crossvector.dbs.chroma import ChromaAdapter
from crossvector.dbs.milvus import MilvusAdapter
from crossvector.dbs.pgvector import PgVectorAdapter

Testing

All tests passing:

• ✅ Unit tests: 62/62 passing
• ✅ Query DSL tests: All compilers validated
• ✅ Engine tests: Updated for new behavior
• ✅ Backend compatibility: Tested with all 4 databases

pytest tests/
# 62 passed

Backend Compatibility Matrix

Feature	AstraDB	ChromaDB	Milvus	PgVector
Metadata-only search	✅	✅	❌	✅
Nested metadata	✅	⚠️ Flattened	✅	✅
All 8 operators	✅	✅	✅	✅
Numeric comparisons	✅	✅	✅	✅ (with casting)
Vector similarity	✅	✅	✅	✅

Documentation

Complete documentation available:

• Query DSL Guide - How to build complex filters
• Architecture - System design patterns
• Database Adapters - Backend-specific features
• API Reference - Complete API documentation

Related Issues

Closes #[issue-number] (if applicable)

Checklist

• Tests added/updated
• Documentation updated
• Breaking changes documented
• All tests passing
• Code formatted with ruff
• Type hints added
• Pre-commit hooks passing