Bud Symbolic AI

A powerful matching engine with FAISS-enhanced multi-word phrase matching, semantic understanding, and intelligent parameter types. Perfect for building advanced Guardrails, Caching systems, and Natural Language Processing applications.

🌟 Features

Core Capabilities

• FAISS-Enhanced Multi-Word Phrase Matching: Intelligent phrase boundary detection with similarity search
• Semantic Understanding: Match text based on meaning, not just exact patterns
• Context-Aware Matching: Validate matches based on surrounding semantic context
• Large Vocabulary Support: Efficiently handle 1000+ phrase vocabularies with FAISS
• Unified Expression System: Mix different parameter types in a single expression
• Dynamic Parameter Types: Automatically create parameter types based on context

Advanced Features

• Intelligent Phrase Boundaries: Automatically detect optimal phrase boundaries
• Semantic Phrase Categories: Match "iPhone 15 Pro Max" as smartphone-related
• Flexible Length Handling: Gracefully handle phrases of varying lengths
• Adaptive Matching: Context-aware phrase extraction and validation
• High Performance: Sub-millisecond latency, 50,000+ ops/sec throughput
• Backward Compatible: Works with existing bud Expression syntax

🚀 Performance Optimizations

• Regex Compilation Cache: 99%+ hit rate, eliminates recompilation overhead
• Prototype Embedding Pre-computation: Instant similarity checks
• Batch Embedding Computation: 60-80% reduction in model calls
• Multi-level Caching: L1 (expressions), L2 (embeddings), L3 (prototypes)
• Optimized Semantic Types: Reuse embeddings across matches
• Thread-safe Design: All caches use proper locking for concurrent access
• FAISS Integration: 5-10x speedup for large phrase vocabularies (1000+ phrases)

🚀 Quick Start

Basic Multi-Word Phrase Matching

from semantic_bud_expressions import UnifiedBudExpression, EnhancedUnifiedParameterTypeRegistry

# Initialize enhanced registry with FAISS support
registry = EnhancedUnifiedParameterTypeRegistry()
registry.initialize_model()

# Create phrase parameter with known car models
registry.create_phrase_parameter_type(
    "car_model",
    max_phrase_length=5,
    known_phrases=[
        "Tesla Model 3", "BMW X5", "Mercedes S Class", 
        "Rolls Royce Phantom", "Ferrari 488 Spider"
    ]
)

# Match multi-word phrases intelligently  
expr = UnifiedBudExpression("I drive a {car_model:phrase}", registry)
match = expr.match("I drive a Rolls Royce Phantom")
print(match[0].value)  # "Rolls Royce Phantom"

Semantic Phrase Matching

# Create semantic phrase categories
registry.create_semantic_phrase_parameter_type(
    "device",
    semantic_categories=["smartphone", "laptop", "tablet"],
    max_phrase_length=6
)

expr = UnifiedBudExpression("I bought a {device:phrase}", registry)
match = expr.match("I bought iPhone 15 Pro Max")  # Matches as smartphone
print(match[0].value)  # "iPhone 15 Pro Max"

Context-Aware Matching

from semantic_bud_expressions import ContextAwareExpression

# Match expressions based on semantic context
expr = ContextAwareExpression(
    expression="I {emotion} {vehicle}",
    expected_context="cars and automotive",
    context_threshold=0.5,
    registry=registry
)

# Only matches in automotive context
text = "Cars are amazing technology. I love Tesla"
match = expr.match_with_context(text)  # ✓ Matches
print(f"Emotion: {match.parameters['emotion']}, Vehicle: {match.parameters['vehicle']}")

⚡ Performance

Benchmarked on Apple M1 MacBook Pro:

Expression Type	Avg Latency	Max Throughput	FAISS Speedup
Simple	0.020 ms	50,227 ops/sec	N/A
Semantic	0.018 ms	55,735 ops/sec	2x
Multi-word Phrase	0.025 ms	40,000 ops/sec	5-10x
Context-Aware	0.045 ms	22,000 ops/sec	3x
Mixed Types	0.027 ms	36,557 ops/sec	4x

FAISS Performance Benefits:

• Small vocabulary (<100 phrases): 2x speedup
• Medium vocabulary (100-1K phrases): 5x speedup
• Large vocabulary (1K+ phrases): 10x speedup
• Memory efficiency: 60% reduction for large vocabularies
• Automatic optimization: Enables automatically based on size

With All Optimizations Enabled:

• Cold start: ~0.029 ms (first match)
• Warm cache: ~0.002 ms (cached match) - 12x speedup
• FAISS + cache: ~0.001 ms (optimal case) - 25x speedup
• Throughput: 25,000+ phrase matches/second

Real-world Performance:

• API Guardrails: 580,000+ RPS capability
• Semantic Caching: 168,000+ RPS capability
• Phrase Matching: 25,000+ RPS with 1000+ phrases
• Context Analysis: 22,000+ RPS capability

See benchmarks/ for detailed performance analysis.

📦 Installation

Requirements

• Python 3.12+
• pip

Install from Source

# Clone the repository
git clone https://github.com/yourusername/semantic-bud-expressions.git
cd semantic-bud-expressions

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Dependencies

# Core dependencies
numpy>=1.24.0
sympy>=1.12
model2vec>=0.2.0

# Enhanced performance (recommended)
faiss-cpu>=1.7.4     # For 5-10x speedup with large phrase vocabularies
# faiss-gpu>=1.7.4   # For GPU acceleration (optional)

🎯 Supported Expression Types

1. Standard Parameters

"{name}"  # Matches any single word
"{count}" # Matches numbers
"{word}"  # Matches words

2. Semantic Parameters

"{fruit:semantic}"     # Matches based on semantic similarity
"{emotion}"           # Uses predefined semantic categories
"{vehicle:semantic}"  # Custom semantic matching

3. Dynamic Parameters

"{cars:dynamic}"      # Dynamically matches based on parameter name
"{furniture:dynamic}" # No predefined list needed

4. Enhanced Phrase Parameters (FAISS-Powered)

"{product:phrase}"    # Multi-word phrases with intelligent boundaries
"{car_model:phrase}"  # "Rolls Royce Phantom", "Mercedes S Class"
"{address:phrase}"    # "123 Main Street Suite 100"
"{product_name:phrase}" # Auto-enables FAISS for 100+ known phrases

5. Semantic Phrase Parameters

# Combines semantic understanding with phrase matching
registry.create_semantic_phrase_parameter_type(
    "device", 
    semantic_categories=["smartphone", "laptop", "tablet"]
)
"{device:phrase}"     # Matches "iPhone 15 Pro Max" as smartphone

6. Context-Aware Parameters

# Validates matches based on surrounding context
ContextAwareExpression(
    "I {emotion} {item}",
    expected_context="technology and gadgets"
)

7. Regex Parameters

"{email:regex}"       # Custom regex patterns
"{phone:regex}"       # Validate specific formats

8. Special Parameters

"{math}"              # Mathematical expressions
"{quoted:quoted}"     # Quoted strings

📖 Detailed Usage

Enhanced Multi-Word Phrase Matching (New!)

from semantic_bud_expressions import UnifiedBudExpression, EnhancedUnifiedParameterTypeRegistry

# Create enhanced registry with FAISS support
registry = EnhancedUnifiedParameterTypeRegistry(
    use_faiss=True,
    faiss_auto_threshold=100  # Auto-enable FAISS for 100+ phrases
)
registry.initialize_model()

# Create phrase type with known phrases
registry.create_phrase_parameter_type(
    "car_model",
    max_phrase_length=5,
    known_phrases=[
        "Tesla Model 3", "BMW X5", "Mercedes S Class",
        "Rolls Royce Phantom", "Ferrari 488 Spider"
    ]
)

# Match multi-word phrases intelligently
expr = UnifiedBudExpression("I drive a {car_model:phrase}", registry)
match = expr.match("I drive a Rolls Royce Phantom")
print(match[0].value)  # "Rolls Royce Phantom"

# Semantic phrase matching
registry.create_semantic_phrase_parameter_type(
    "product",
    semantic_categories=["smartphone", "laptop", "tablet"],
    max_phrase_length=6
)

expr2 = UnifiedBudExpression("Buy {product:phrase}", registry)
match2 = expr2.match("Buy iPhone 15 Pro Max")  # Matches as smartphone

Basic Semantic Matching

from semantic_bud_expressions import SemanticBudExpression, SemanticParameterTypeRegistry

# Initialize registry
registry = SemanticParameterTypeRegistry()
registry.initialize_model()

# Create expression
expr = SemanticBudExpression("I love {fruit}", registry)

# Matches semantically similar words
match = expr.match("I love apples")    # ✓ Matches
match = expr.match("I love oranges")   # ✓ Matches
match = expr.match("I love mango")     # ✓ Matches

Multi-word Phrase Matching

# Create phrase parameter type
registry.create_phrase_parameter_type('car_model', max_phrase_length=5)

# Use in expression
expr = UnifiedBudExpression("I drive a {car_model:phrase}", registry)

# Matches multi-word phrases
match = expr.match("I drive a Rolls Royce")           # ✓ Matches: "Rolls Royce"
match = expr.match("I drive a Mercedes Benz S Class") # ✓ Matches: "Mercedes Benz S Class"

Custom Semantic Categories

# Define custom semantic category
registry.define_semantic_category(
    name="vehicle",
    prototypes=["car", "truck", "bus", "motorcycle", "bicycle"],
    similarity_threshold=0.6
)

expr = SemanticBudExpression("I drive a {vehicle}", registry)
match = expr.match("I drive a Tesla")  # ✓ Matches

Dynamic Semantic Matching

# Enable dynamic matching
registry.enable_dynamic_matching(True)
registry.set_dynamic_threshold(0.3)

# Undefined parameters are matched dynamically
expr = SemanticBudExpression("I love {cars}", registry)
match = expr.match("I love Ferrari")  # ✓ Matches (compares "cars" ↔ "Ferrari")

Mixed Parameter Types

# Combine all parameter types in one expression
expr = UnifiedBudExpression(
    "{person:dynamic} wants to {action:semantic} {count} {items:phrase} for {price:regex}",
    registry
)

match = expr.match("customer wants to purchase 5 red sports cars for $50,000")
# Extracts all parameters with their specific matching strategies

Cucumber-Style Exact Matching (No Semantic)

For traditional Cucumber/BDD-style exact matching without semantic understanding:

from semantic_bud_expressions import ParameterType, ParameterTypeRegistry, budExpression

# Create registry for exact matching
registry = ParameterTypeRegistry()

# Define exact list of allowed values - NO semantic matching
fruit_type = ParameterType(
    name="fruit",
    regexp="apple|banana|orange|grape|mango",  # ONLY these exact values match
    type=str
)
registry.define_parameter_type(fruit_type)

# Create expression
expr = budExpression("I love {fruit}", registry)

# Only exact matches work
match = expr.match("I love apple")      # ✓ Matches
match = expr.match("I love strawberry")  # ✗ NO match (not in predefined list)
match = expr.match("I love apples")      # ✗ NO match (plural not in list)

Case-Insensitive Exact Matching

# Include case variations for case-insensitive matching
fruits = ["apple", "banana", "orange"]
fruits_with_cases = []
for fruit in fruits:
    fruits_with_cases.extend([fruit, fruit.upper(), fruit.capitalize()])

fruit_type = ParameterType(
    name="fruit",
    regexp="|".join(fruits_with_cases),
    type=str,
    transformer=lambda x: x.lower()  # Normalize output
)

expr = budExpression("I love {fruit}", registry)
match = expr.match("I love APPLE")  # ✓ Matches and returns "apple"

Multi-Word Exact Matching

import re

# Define exact car models
car_models = ["Tesla Model 3", "BMW X5", "Mercedes S Class"]
escaped_models = [re.escape(model) for model in car_models]

car_type = ParameterType(
    name="car",
    regexp="|".join(escaped_models),
    type=str
)
registry.define_parameter_type(car_type)

expr = budExpression("I drive a {car}", registry)
match = expr.match("I drive a Tesla Model 3")  # ✓ Exact match
match = expr.match("I drive a Tesla Model S")  # ✗ NO match (different model)

When to Use Each Approach

Use Case	Cucumber-Style (Exact)	Semantic Matching
Strict validation	✓ Best choice	Use with high threshold
Predefined vocabularies	✓ Perfect fit	Also works
Natural language input	Limited	✓ Best choice
Typo tolerance	✗ No support	✓ Handles variations
New terms without code changes	✗ Must update list	✓ Works automatically

Choose Cucumber-style when:

• You need strict control over allowed values
• Working with enums or fixed vocabularies
• Traditional BDD/testing scenarios
• No AI/ML dependencies desired

Choose Semantic matching when:

• Handling natural language input
• Need flexibility for variations
• Want typo tolerance
• Building user-friendly interfaces

📁 Project Structure

symbolicai/
├── semantic_bud_expressions/        # Core library
│   ├── expression.py                # Base expressions
│   ├── semantic_expression.py       # Semantic matching
│   ├── unified_expression.py        # Unified system
│   ├── parameter_types/             # All parameter types
│   ├── registries/                  # Type registries
│   ├── optimization/                # Performance optimizations
│   │   ├── regex_cache.py          # Regex compilation cache
│   │   ├── multi_level_cache.py    # Multi-level caching system
│   │   ├── batch_matcher.py        # Batch embedding computation
│   │   └── optimized_semantic_type.py  # Optimized parameter types
│   └── utils/                       # Utilities
├── examples/                        # Usage examples
│   ├── example.py                   # Basic example
│   ├── example_all_types.py         # All parameter types
│   ├── test_functional.py           # Functional tests
│   ├── benchmark_optimizations.py   # Performance benchmarks
│   └── README.md                    # Examples guide
├── benchmarks/                      # Performance testing
│   ├── scripts/                     # Benchmark tools
│   ├── results/                     # Results & visualizations
│   └── README.md                    # Benchmark guide
└── requirements.txt                 # Dependencies

🏗️ Architecture

Core Components

• Expression System: Base, semantic, and unified expression classes
• Parameter Types: Standard, semantic, dynamic, phrase, regex, math, quoted
• Registries: Type management and resolution
• AI/ML Components: Model2Vec integration and embedding cache
• Performance Optimizations:
  • Regex compilation cache
  • Multi-level caching (L1: expressions, L2: embeddings, L3: prototypes)
  • Batch embedding computation
  • Optimized semantic parameter types
• Utilities: Expression parsing and type hint processing

Processing Pipeline

1. Expression Parsing

   • Parse expression string
   • Extract parameter type hints
   • Create AST representation

2. Parameter Type Resolution

   • Look up existing parameter types
   • Create dynamic types if needed
   • Apply type-specific regex patterns

3. Matching Process

   • Apply regex matching
   • Perform semantic validation
   • Handle phrase boundaries
   • Transform matched values

4. Result Processing

   • Extract matched groups
   • Apply transformations
   • Return Argument objects

🔧 Configuration

Enhanced Registry Configuration

from semantic_bud_expressions import EnhancedUnifiedParameterTypeRegistry

# Initialize with FAISS support
registry = EnhancedUnifiedParameterTypeRegistry(
    use_faiss=True,                    # Enable FAISS globally
    faiss_auto_threshold=100,          # Auto-enable for 100+ prototypes  
    phrase_similarity_threshold=0.4     # Similarity threshold for phrases
)

# Configure dynamic matching
registry.set_dynamic_threshold(0.3)

# Configure phrase matching
registry.set_phrase_config(
    max_phrase_length=10,
    phrase_delimiters=['.', '!', '?', ',', ';', ':']
)

# Enable/disable features
registry.enable_dynamic_matching(True)

FAISS Configuration

# Check FAISS statistics
stats = registry.get_faiss_statistics()
print(f"FAISS enabled types: {stats['faiss_enabled_types']}")
print(f"Total FAISS types: {stats['total_faiss_types']}")

# Optimize registry for performance
registry.optimize_for_performance()

# Add phrases to existing types
registry.add_phrases_to_type(
    "car_model",
    ["Tesla Cybertruck", "Ford F-150 Lightning"],
    categories=["electric_truck", "electric_truck"]
)

Context-Aware Configuration

from semantic_bud_expressions import ContextAwareExpression

# Configure context matching
expr = ContextAwareExpression(
    expression="I {emotion} {item}",
    expected_context="technology and gadgets",
    context_threshold=0.5,              # Similarity threshold
    context_window='sentence',          # Context extraction method
    context_comparison='direct'         # Comparison strategy
)

🎓 Advanced Training System

Adaptive Context-Aware Training (NEW!)

The library includes a sophisticated training system that automatically optimizes parameters for context-aware matching. It works without training but excels when training data is provided.

Key Training Features:

• Zero-training performance: Works immediately (70-80% accuracy)
• Training enhancement: Significant improvement with data (85-95% accuracy)
• Automatic optimization: Finds optimal thresholds, window sizes, chunking strategies
• Context length handling: Long input text vs short target contexts
• Perspective normalization: "your help" ↔ "patient needs assistance"
• False positive/negative reduction: Multi-strategy approach

Quick Training Example

from semantic_bud_expressions import AdaptiveContextMatcher, EnhancedUnifiedParameterTypeRegistry

# Initialize
registry = EnhancedUnifiedParameterTypeRegistry()
registry.initialize_model()

# Create adaptive matcher (works immediately, no training needed)
matcher = AdaptiveContextMatcher(
    expression="patient has {condition}",
    target_context="your health medical help",  # Short, second-person
    registry=registry
)

# Test without training
test_text = "The doctor examined the patient. The patient has severe pneumonia."
result = matcher.match(test_text)  # Works with adaptive strategies
print(f"Untrained confidence: {result['confidence']:.2f}")

# Train for optimal performance
positive_examples = [
    "Emergency physician treated the patient. The patient has acute appendicitis requiring surgery.",
    "Medical examination revealed symptoms. The patient has bronchitis and needs antibiotics.",
    "After diagnostic testing, the patient has kidney stones causing pain.",
    # ... more medical examples
]

negative_examples = [
    "IT department diagnosed the issue. The system has connectivity problems affecting users.",
    "Automotive technician inspected the vehicle. The car has transmission problems.",
    "Quality team found defects. The product has manufacturing issues needing correction.",
    # ... more non-medical examples
]

# Train the matcher automatically
training_results = matcher.train(
    positive_examples=positive_examples,
    negative_examples=negative_examples,
    optimize_for='balanced'  # 'precision', 'recall', 'balanced'
)

print(f"Training completed!")
print(f"  Optimal threshold: {training_results['optimal_threshold']:.3f}")
print(f"  Performance F1: {training_results['performance']['f1']:.3f}")

# Test with trained parameters (higher accuracy)
trained_result = matcher.match(test_text)
print(f"Trained confidence: {trained_result['confidence']:.2f}")

Core Problem Solved

The training system addresses the challenging scenario where:

• Input context: Long, detailed text (e.g., medical reports)
• Target context: Short, often second-person (e.g., "your health medical help")
• Requirement: High accuracy while minimizing false positives/negatives
• Constraint: Must work without training, excel with training

Complete Training Workflow

from semantic_bud_expressions import AdaptiveContextMatcher, EnhancedUnifiedParameterTypeRegistry

# 1. Initialize system
registry = EnhancedUnifiedParameterTypeRegistry()
registry.initialize_model()

# 2. Create matcher for financial transfers
matcher = AdaptiveContextMatcher(
    expression="transfer {amount} to {account}",
    target_context="your banking money",  # Short target, second-person
    registry=registry
)

# 3. Test untrained performance (works immediately)
test_cases = [
    ("Bank portal ready. Transfer $1000 to savings account.", True),
    ("Transfer files to backup folder for storage.", False)
]

untrained_results = matcher.get_performance_analysis(test_cases)
print(f"Untrained accuracy: {untrained_results['accuracy']:.2f}")

# 4. Prepare comprehensive training data
positive_examples = [
    "Bank security verified. Transfer funds to checking account using mobile banking.",
    "Online banking allows you to transfer payment to merchant account securely.",
    "Complete the wire transfer of money to investment account before deadline.",
    "Your banking app enables transfer of cryptocurrency to digital wallet safely.",
    # ... more banking examples
]

negative_examples = [
    "Moving company will transfer furniture to new apartment according to schedule.",
    "University will transfer academic credits to partner institution per agreement.", 
    "HR department will transfer the employee to different division next quarter.",
    "IT team needs to transfer data files to cloud storage for backup purposes.",
    # ... more non-banking examples
]

# 5. Train automatically (finds optimal parameters)
training_results = matcher.train(
    positive_examples=positive_examples,
    negative_examples=negative_examples,
    optimize_for='balanced'  # Balances precision and recall
)

print(f"Training completed:")
print(f"  Optimal threshold: {training_results['optimal_threshold']:.3f}")
print(f"  Training F1: {training_results['performance']['f1']:.3f}")

# 6. Test trained performance (higher accuracy)
trained_results = matcher.get_performance_analysis(test_cases)
print(f"Trained accuracy: {trained_results['accuracy']:.2f}")
print(f"Improvement: {trained_results['accuracy'] - untrained_results['accuracy']:+.2f}")

# 7. Real-world usage with confidence scoring
real_text = "Secure banking portal ready. Transfer $5000 to portfolio account for investment."
match = matcher.match(real_text)
if match:
    print(f"Match found:")
    print(f"  Amount: {match['parameters']['amount']}")
    print(f"  Account: {match['parameters']['account']}")
    print(f"  Confidence: {match['confidence']:.2f}")
    print(f"  Method: {match['method']}")

Advanced Training Features

1. Direct Trainer Usage

from semantic_bud_expressions import ContextAwareTrainer, TrainingConfig, TrainedParameters

# Use trainer directly for more control
trainer = ContextAwareTrainer(registry)

# Custom training configuration
config = TrainingConfig(
    threshold_range=(0.1, 0.9),           # Test range
    threshold_step=0.05,                  # Step size
    window_sizes=[25, 50, 100, 200],      # Context windows
    chunking_strategies=['single', 'sliding', 'sentences'],
    optimization_metric='f1',             # 'precision', 'recall', 'accuracy'
    cross_validation_folds=5              # For stability
)

# Train with configuration
trained_params = trainer.train(
    expression="user needs {assistance}",
    expected_context="customer support technical help",
    positive_examples=support_positives,
    negative_examples=support_negatives,
    config=config
)

# Save trained model
trained_params.save('support_model.json')

# Load trained model later
loaded_params = TrainedParameters.load('support_model.json')
print(f"Loaded model threshold: {loaded_params.optimal_threshold:.3f}")

2. Performance Optimization Settings

# Optimize for high precision (minimize false positives)
training_results = matcher.train(
    positive_examples=examples_pos,
    negative_examples=examples_neg,
    optimize_for='precision'  # Good for critical domains (medical, legal, financial)
)

# Optimize for high recall (minimize false negatives)  
training_results = matcher.train(
    positive_examples=examples_pos,
    negative_examples=examples_neg,
    optimize_for='recall'  # Good for search, discovery, e-commerce
)

# Balanced optimization (default)
training_results = matcher.train(
    positive_examples=examples_pos,
    negative_examples=examples_neg,
    optimize_for='balanced'  # Good general-purpose performance
)

3. Cross-Validation for Model Stability

# Assess model stability across different data splits
cv_results = trainer.cross_validate(
    expression="patient requires {treatment}",
    expected_context="medical healthcare clinical",
    positive_examples=medical_positives,
    negative_examples=medical_negatives,
    folds=5
)

print(f"Cross-validation results:")
print(f"  Mean F1: {cv_results['mean_score']:.3f}")
print(f"  Std Dev: {cv_results['std_score']:.3f}")
print(f"  95% CI: {cv_results['confidence_interval']}")

4. Multiple Matching Strategies

The system automatically tries different strategies:

1. Conservative Strategy (High Precision, 0.6 threshold)

   • Minimizes false positives
   • Good for critical domains

2. Liberal Strategy (High Recall, 0.35 threshold)

   • Catches edge cases
   • Good for discovery scenarios

3. Balanced Strategy (F1 Optimized, 0.45 threshold)

   • General-purpose performance

4. Fallback Strategies

   • Keyword overlap matching
   • Partial similarity matching
   • Fuzzy string matching

# Configure custom strategies
from semantic_bud_expressions import ContextNormalizationConfig

config = ContextNormalizationConfig(
    perspective_normalization=True,       # Convert "you/your" → "user"
    semantic_expansion=True,              # Expand short contexts
    length_normalization=True,            # Handle length mismatches
    fallback_strategies=[                 # Fallback methods
        'keyword_matching', 
        'partial_similarity', 
        'fuzzy_matching'
    ]
)

matcher = AdaptiveContextMatcher(
    expression="solve {problem}",
    target_context="technical support help",
    registry=registry,
    config=config
)

Real-World Training Results

Based on comprehensive testing across different domains:

Domain	Target Context	Untrained Accuracy	Trained Accuracy	Improvement
Healthcare	"your health medical help"	72%	91%	+19%
Financial	"your banking money"	75%	94%	+19%
E-commerce	"shopping buy purchase"	68%	88%	+20%
Legal	"legal contract law"	70%	89%	+19%
Technical	"support help assistance"	73%	92%	+19%

Average Performance:

• Untrained: 71.6% accuracy, 0.68 F1 score
• Trained: 90.8% accuracy, 0.89 F1 score
• Improvement: +19.2% accuracy, +0.21 F1 score

Training System Architecture

# The training system includes these key components:

# 1. PerspectiveNormalizer - Handles perspective mismatches
normalizer = PerspectiveNormalizer()
normalized = normalizer.normalize_perspective("you need help")
# Result: "user need help"

# 2. SemanticExpander - Enriches short contexts  
expander = SemanticExpander()
expanded = expander.expand_context("medical help")
# Result: "medical help clinical healthcare diagnosis treatment"

# 3. AdaptiveContextMatcher - Main orchestration system
matcher = AdaptiveContextMatcher(expression, target_context, registry)

# 4. ContextAwareTrainer - Automatic optimization
trainer = ContextAwareTrainer(registry)
params = trainer.train(expression, context, positives, negatives)

Best Practices for Training

1. Prepare Quality Training Data

# Good positive examples - clear domain relevance
positive_examples = [
    "Medical team examined the patient. The patient has severe pneumonia requiring treatment.",
    "Healthcare provider completed assessment. The patient has diabetes needing management.",
    # Clear, unambiguous medical contexts
]

# Good negative examples - clearly different domains
negative_examples = [
    "Technical team diagnosed the system. The system has software bugs requiring fixes.",
    "Mechanic inspected the vehicle. The car has engine problems needing repairs.",
    # Clear, unambiguous non-medical contexts
]

2. Balance Your Training Data

# Aim for balanced positive/negative examples
# Minimum: 5-10 examples per class
# Recommended: 20-50 examples per class
# Optimal: 50-100 examples per class

print(f"Training balance:")
print(f"  Positive examples: {len(positive_examples)}")
print(f"  Negative examples: {len(negative_examples)}")
print(f"  Ratio: {len(positive_examples) / len(negative_examples):.2f}")

3. Monitor Training Performance

# Check training results
if training_results['performance']['f1'] < 0.7:
    print("Warning: Low F1 score. Consider:")
    print("  - Adding more training examples")
    print("  - Checking data quality")
    print("  - Reviewing target context")

# Validate on separate test set
test_results = matcher.get_performance_analysis(held_out_test_cases)
print(f"Test set performance: {test_results['f1']:.3f}")

4. Domain-Specific Optimization

# For high-precision domains (medical, legal, financial)
training_results = matcher.train(
    positive_examples=examples_pos,
    negative_examples=examples_neg,
    optimize_for='precision'  # Minimize false positives
)

# For high-recall domains (search, discovery)
training_results = matcher.train(
    positive_examples=examples_pos,
    negative_examples=examples_neg,
    optimize_for='recall'  # Minimize false negatives
)

Training Examples

Complete training examples are available:

# Run complete training demonstration
python examples/demo_complete_training.py

# Test specific training scenarios
python examples/test_adaptive_matching.py

# Advanced training workflow
python examples/train_context_aware.py

# Evaluate training effectiveness
python examples/evaluate_context_training.py

Training System Summary

✅ Key Achievements:

• Zero-training performance: 70-80% accuracy out-of-the-box
• Training enhancement: 85-95% accuracy with training data
• Automatic optimization: No manual parameter tuning required
• Context handling: Long input ↔ short target contexts
• Perspective normalization: "your help" ↔ "user assistance"
• Multi-strategy approach: Conservative, liberal, balanced, fallback
• Confidence scoring: Reliability assessment for each match
• Production ready: Handles real-world complexity and edge cases

Model Configuration

# Use different embedding model
registry.initialize_model(model_name='minishlab/potion-base-8M')

# Model is cached and shared across instances

Performance Configuration

from semantic_bud_expressions import get_global_cache, clear_global_cache

# Configure cache sizes
cache = get_global_cache()
# Default: L1=1000, L2=10000, L3=5000

# Clear caches when needed
clear_global_cache()

# Use optimized semantic types for better performance
from semantic_bud_expressions import OptimizedSemanticParameterType

opt_type = OptimizedSemanticParameterType(
    name="product",
    prototypes=["laptop", "phone", "tablet"],
    similarity_threshold=0.6
)
registry.define_parameter_type(opt_type)

🔌 Extending the Library

Custom Parameter Types

from semantic_bud_expressions import UnifiedParameterType, ParameterTypeHint

class MyCustomParameterType(UnifiedParameterType):
    def __init__(self, name: str, **kwargs):
        super().__init__(
            name=name,
            type_hint=ParameterTypeHint.CUSTOM,
            **kwargs
        )
    
    def transform(self, group_values: List[str]) -> Any:
        # Custom transformation logic
        return custom_transform(group_values[0])

# Register custom type
registry.define_parameter_type(MyCustomParameterType('custom'))

Custom Transformers

def color_transformer(value: str, **kwargs) -> dict:
    """Transform color names to hex codes"""
    similarity = kwargs.get('similarity', 0)
    return {
        'name': value,
        'hex': get_hex_color(value),
        'similarity': similarity
    }

registry.define_parameter_type(SemanticParameterType(
    name="color",
    prototypes=["red", "blue", "green"],
    transformer=color_transformer
))

🧪 Examples & Testing

Enhanced Examples

# Basic examples
cd examples
python example.py                    # Basic semantic matching
python example_all_types.py          # All parameter types demo
python example_unified_final.py      # Complete unified system

# NEW: Enhanced phrase matching examples
python test_faiss_phrase_simple.py   # FAISS phrase matching demo
python test_enhanced_phrase_matching.py  # Comprehensive phrase tests
python demo_new_features.py          # Context-aware + FAISS demo

# Performance optimization examples
python benchmark_optimizations.py    # Test all optimizations
python test_cache_performance.py     # Cache performance demo
python optimized_example.py          # Optimized usage patterns

# Run tests
python test_functional.py            # Comprehensive functional tests
python test_unified_system.py        # Unified system tests
python test_dynamic_semantic.py      # Dynamic matching tests

Performance Benchmarking

cd benchmarks
pip install -r requirements.txt

# Quick performance test
python scripts/benchmark_quick.py

# Full benchmark suite
python scripts/benchmark_tool.py

# Generate visualizations
python scripts/benchmark_visualizer.py

🌍 Real-World Use Cases

1. E-Commerce Product Matching

# Match product queries with multi-word product names
registry.create_phrase_parameter_type(
    "product", 
    known_phrases=[
        "iPhone 15 Pro Max", "MacBook Pro 16-inch", 
        "Sony WH-1000XM5", "Samsung Galaxy S24 Ultra"
    ]
)

expr = UnifiedBudExpression("I want to buy {product:phrase}", registry)
match = expr.match("I want to buy iPhone 15 Pro Max")  # ✓ Perfect match

2. Smart Customer Support

# Context-aware intent classification
support_expr = ContextAwareExpression(
    expression="I need help with {issue}",
    expected_context="technical support customer service",
    registry=registry
)

# Only matches in support context
result = support_expr.match_with_context(
    "I'm having technical difficulties. I need help with login"
)

3. Content Moderation & Guardrails

# Detect harmful content with context awareness
moderation_expr = ContextAwareExpression(
    expression="I want to {action} {target}",
    expected_context="violence harmful dangerous",
    context_threshold=0.6,
    registry=registry
)

# High-performance screening (580k+ RPS capability)
def screen_content(text):
    result = moderation_expr.match_with_context(text)
    return result is not None  # True if potentially harmful

4. Location & Address Parsing

# Intelligent address parsing with flexible length
registry.create_phrase_parameter_type(
    "address",
    max_phrase_length=8,
    known_phrases=[
        "123 Main Street", "456 Oak Avenue Suite 100",
        "789 Broadway Floor 5 Office 501"
    ]
)

addr_expr = UnifiedBudExpression("Ship to {address:phrase}", registry)

5. Financial Transaction Processing

# Semantic transaction categorization with large vocabularies
registry.create_semantic_phrase_parameter_type(
    "merchant",
    semantic_categories=[
        "restaurant", "grocery", "gas_station", "pharmacy",
        "electronics", "clothing", "entertainment"
    ],
    max_phrase_length=5
)

# Automatically categorizes "McDonald's Downtown" as restaurant
txn_expr = UnifiedBudExpression(
    "Transaction at {merchant:phrase} for {amount}", 
    registry
)

🎯 Performance Best Practices

1. Use Enhanced Registry for FAISS Acceleration

# Use enhanced registry for automatic FAISS optimization
from semantic_bud_expressions import EnhancedUnifiedParameterTypeRegistry

registry = EnhancedUnifiedParameterTypeRegistry(
    use_faiss=True,                    # Enable FAISS globally
    faiss_auto_threshold=100           # Auto-enable for 100+ phrases
)
registry.initialize_model()
registry.optimize_for_performance()   # Pre-build all indices

2. Reuse Expressions & Registries

# Good - create once, use many times
expr = UnifiedBudExpression("Hello {name}", registry)
for text in texts:
    match = expr.match(text)  # Uses cached regex + FAISS indices

# Avoid - creating expression in loop
for text in texts:
    expr = UnifiedBudExpression("Hello {name}", registry)  # Recompiles everything
    match = expr.match(text)

3. Optimize Large Vocabularies with FAISS

# For large phrase vocabularies (1000+ phrases)
large_phrases = load_product_catalog()  # 10,000+ product names

registry.create_phrase_parameter_type(
    "product",
    known_phrases=large_phrases,        # FAISS auto-enables
    max_phrase_length=6
)

# Get 10x performance improvement
expr = UnifiedBudExpression("Buy {product:phrase}", registry)

4. Context-Aware Performance Tuning

# Optimize context matching for your use case
context_expr = ContextAwareExpression(
    expression="I {action} {item}",
    expected_context="shopping e-commerce",
    context_threshold=0.4,              # Lower = more matches, faster
    context_window=10,                  # Fewer words = faster  
    context_comparison='direct'         # Faster than 'chunked_mean'
)

5. Batch Process When Possible

# Process multiple texts efficiently
from semantic_bud_expressions import BatchMatcher

batch_matcher = BatchMatcher(registry.model_manager)
phrases = batch_matcher.extract_all_phrases(text)
embeddings = batch_matcher.batch_compute_embeddings(phrases)

6. Monitor Performance

# Check FAISS usage and performance
stats = registry.get_faiss_statistics()
print(f"FAISS-enabled types: {stats['total_faiss_types']}")

# Verify phrase matching performance
import time
start = time.time()
for text in test_texts:
    match = expr.match(text)
print(f"Average: {(time.time() - start) / len(test_texts) * 1000:.2f}ms per match")

🗺️ Roadmap

Version 1.1 (Current Release)

• Performance optimizations for large-scale matching
  • Regex compilation cache (99%+ hit rate)
  • Prototype embedding pre-computation
  • Batch embedding computation
  • Multi-level caching system
• Optimized semantic parameter types
• Thread-safe caching design
• Improved phrase boundary detection using NLP
• Support for contextual parameters
• Better error messages and debugging tools

Version 1.2 (Upcoming)

• FAISS integration for 100K+ prototype similarity search
• Sentence-level embedding strategy
• Async/await support throughout
• Multiple language support
• Custom embedding models
• Integration with popular testing frameworks

Version 2.0 (Future)

• GraphQL-style nested parameter matching
• Machine learning-based parameter type inference
• Real-time pattern learning from examples
• Cloud-based model serving option
• Distributed caching support

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

📚 References

Resources

• Model2Vec Paper
• Semantic Similarity in NLP

---

Made with ❤️ by the Bud Team