A Python implementation of the VETTING (Verification and Evaluation Tool for Targeting Invalid Narrative Generation) framework for LLM safety and educational applications.
Developed by VIABLE Lab at the University of Florida.
The VETTING framework implements a dual-LLM architecture that separates conversational logic from policy enforcement, preventing prompt injection attacks on safety rules and enabling verifiable policy compliance. This architectural approach is particularly effective for educational applications where you need to guide learning without revealing direct answers.
- 🛡️ Architectural Policy Isolation: Complete separation between user interaction (Chat-Layer) and policy enforcement (Verification-Layer)
- 🔄 Iterative Verification Loop: Automatic refinement when responses don't meet verification criteria
- 🏫 Educational Focus: Specialized support for tutoring and homework help scenarios
- 🌐 Multi-Provider Support: Works with OpenAI, Anthropic Claude, and Google Gemini
- 💰 Cost Tracking: Comprehensive cost monitoring and analysis
- ⚙️ Flexible Configuration: Environment variables, config files, or programmatic setup
- 🔍 Safety Features: Built-in safety prefix detection and content filtering
┌─────────────────┐ ┌─────────────────┐
│ User Input │ │ Chat-Layer │
│ │───▶│ (LLM-A) │
│ │ │ │
└─────────────────┘ └─────────┬───────┘
│
▼
┌─────────────────┐
│ Verification- │
│ Layer │◀─── Confidential
│ (LLM-B) │ Policy
└─────────┬───────┘
│
▼
┌─────────────────┐
│ Pass/Fail + │
│ Feedback │
└─────────────────┘
pip install vetting-python✅ Now available on PyPI! View the package: https://pypi.org/project/vetting-python/
Get started in seconds! Click the Colab badge above to try VETTING framework with interactive examples - no installation required!
git clone https://github.com/hichipli/vetting-python.git
cd vetting-python
pip install -e .pip install aiohttp pydantic dataclasses-jsonOptional dependencies:
pip install PyYAML # For YAML configuration files🚀 Want to try immediately?
- Interactive demo with examples!
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-claude-api-key" # Optional
export GOOGLE_API_KEY="your-gemini-api-key" # Optionalimport asyncio
from vetting_python import VettingFramework, VettingConfig, ChatMessage, OpenAIProvider
async def basic_example():
# Setup provider
provider = OpenAIProvider(api_key="your-api-key")
# Create framework
framework = VettingFramework(chat_provider=provider)
# Simple chat configuration
config = VettingConfig(
mode="chat",
chat_model={"model_id": "gpt-4o-mini", "temperature": 0.7}
)
# Create conversation
messages = [ChatMessage("user", "Explain photosynthesis in simple terms.")]
# Process
response = await framework.process(messages, config)
print(f"Response: {response.content}")
print(f"Cost: ${response.total_cost:.4f}")
# Run the example
asyncio.run(basic_example())import asyncio
from vetting_python import VettingFramework, OpenAIProvider
from vetting_python.config import VettingConfigBuilder
async def educational_example():
provider = OpenAIProvider(api_key="your-api-key")
framework = VettingFramework(chat_provider=provider)
# Educational configuration with answer key
config = (VettingConfigBuilder()
.vetting_mode()
.chat_model("gpt-4o-mini")
.verification_model("gpt-4o-mini")
.add_context_item(
question_text="What is the capital of France?",
correct_answer="Paris",
key_concepts=["Paris", "France", "capital city"]
)
.build())
# Student asks directly for the answer
messages = [ChatMessage("user", "What is the capital of France? I need this for homework.")]
# Process with verification
response = await framework.process(messages, config)
print(f"Response: {response.content}")
print(f"Verification passed: {response.verification_passed}")
print(f"Attempts made: {response.attempt_count}")
asyncio.run(educational_example())The framework supports comprehensive configuration through environment variables:
# API Keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="..."
# Default Models
export VETTING_DEFAULT_CHAT_MODEL="gpt-4o-mini"
export VETTING_DEFAULT_VERIFICATION_MODEL="gpt-4o-mini"
export VETTING_DEFAULT_PROVIDER="openai"
# Generation Parameters
export VETTING_TEMPERATURE_CHAT="0.7"
export VETTING_TEMPERATURE_VERIFICATION="0.1"
export VETTING_MAX_TOKENS_CHAT="1024"
export VETTING_MAX_TOKENS_VERIFICATION="512"
export VETTING_MAX_ATTEMPTS="3"
# Features
export VETTING_ENABLE_SAFETY_PREFIX="true"
export VETTING_ENABLE_EDUCATIONAL_RULES="true"
export VETTING_ENABLE_COST_TRACKING="true"
# Logging
export VETTING_LOG_LEVEL="INFO"
export VETTING_LOG_REQUESTS="false"You can also use JSON or YAML configuration files:
{
"providers": {
"openai": {
"provider_type": "openai",
"api_key": "your-key",
"timeout": 60,
"max_retries": 3
}
},
"default_provider": "openai",
"default_chat_model": "gpt-4o-mini",
"default_verification_model": "gpt-4o-mini",
"enable_safety_prefix": true,
"enable_educational_rules": true
}from vetting_python.config import VettingSettings
# Load from file
settings = VettingSettings.from_file("config.json")
# Load from environment
settings = VettingSettings.from_env()
# Create provider and framework
provider = settings.get_provider_instance("openai")
framework = VettingFramework(chat_provider=provider)from vetting_python import OpenAIProvider, ClaudeProvider
# Use different providers for chat and verification
chat_provider = OpenAIProvider(api_key="openai-key")
verification_provider = ClaudeProvider(api_key="claude-key")
framework = VettingFramework(
chat_provider=chat_provider,
verification_provider=verification_provider
)
config = VettingConfig(
mode="vetting",
chat_model={"model_id": "gpt-4o-mini"},
verification_model={"model_id": "claude-3-haiku"}
)from vetting_python.utils import CostTracker
# Setup cost tracking
cost_tracker = CostTracker(enable_persistence=True)
# After processing requests
cost_tracker.track_response(response, "openai", provider, provider)
# Get cost summary
summary = cost_tracker.get_summary()
print(f"Total cost: ${summary.total_cost:.4f}")
print(f"Total tokens: {summary.total_tokens}")
# Print detailed breakdown
cost_tracker.print_summary()config = (VettingConfigBuilder()
.vetting_mode()
.chat_model("gpt-4o-mini", temperature=0.8)
.verification_model("gpt-4o-mini", temperature=0.1)
.chat_system_prompt(
"You are a Socratic tutor. Guide students through discovery "
"rather than giving direct answers. Always end with a question."
)
# Multiple context items
.add_context_item(
question_text="What is photosynthesis?",
subject="Biology",
correct_answer="The process by which plants convert light energy into chemical energy",
key_concepts=["photosynthesis", "chlorophyll", "glucose", "oxygen"],
explanation="Plants use sunlight, CO2, and water to produce glucose and oxygen"
)
.add_context_item(
question_text="What gas do plants absorb during photosynthesis?",
subject="Biology",
correct_answer="Carbon dioxide",
key_concepts=["carbon dioxide", "CO2"]
)
.safety_features(enable_educational_rules=True)
.session_info(session_id="tutoring_001", user_id="student_123")
.build())from vetting_python.utils import ValidationUtils
# Validate configuration
validation = ValidationUtils.validate_vetting_config(config)
if not validation["valid"]:
print(f"Config errors: {validation['issues']}")
# Validate messages
validation = ValidationUtils.validate_messages(messages)
if validation["warnings"]:
print(f"Message warnings: {validation['warnings']}")
# Validate API key format
validation = ValidationUtils.validate_api_key(api_key, "openai")
if not validation["valid"]:
print(f"API key issues: {validation['issues']}")The main framework class that orchestrates the dual-LLM architecture.
VettingFramework(
chat_provider: Provider,
verification_provider: Optional[Provider] = None
)Methods:
async process(messages: List[ChatMessage], config: VettingConfig) -> VettingResponse
Configuration object for the vetting process.
VettingConfig(
mode: Literal["chat", "vetting"] = "vetting",
chat_model: ModelConfig,
verification_model: Optional[ModelConfig] = None,
max_attempts: int = 3,
chat_system_prompt: Optional[str] = None,
verification_system_prompt: Optional[str] = None,
context_items: Optional[List[ContextItem]] = None,
session_id: Optional[str] = None,
user_id: Optional[str] = None,
enable_safety_prefix: bool = True,
enable_educational_rules: bool = True
)Response object containing the result and metadata.
@dataclass
class VettingResponse:
content: str
mode: Literal["chat", "vetting"]
requires_attention: bool = False
verification_passed: Optional[bool] = None
attempt_count: int = 1
stop_reason: Optional[StopReason] = None
attempts: Optional[List[AttemptDetail]] = None
chat_usage: Optional[Usage] = None
verification_usage: Optional[Usage] = None
total_usage: Optional[Usage] = None
total_cost: float = 0.0
processing_time_ms: Optional[float] = None
# ... additional metadata fieldsThe VettingConfigBuilder provides a fluent API for building configurations:
config = (VettingConfigBuilder()
.vetting_mode() # or .chat_mode()
.chat_model("gpt-4o-mini", temperature=0.7, max_tokens=1024)
.verification_model("gpt-4o-mini", temperature=0.1, max_tokens=512)
.max_attempts(3)
.add_context_item(
question_text="What is X?",
correct_answer="Y",
key_concepts=["concept1", "concept2"]
)
.safety_features(enable_safety_prefix=True, enable_educational_rules=True)
.session_info(session_id="session_123", user_id="user_456")
.build())OpenAIProvider(
api_key: str,
base_url: str = "https://api.openai.com/v1",
max_retries: int = 3,
timeout: int = 60,
organization: Optional[str] = None
)Supported Models (2025 Pricing):
gpt-4.1,gpt-4.1-mini,gpt-4.1-nanogpt-4o,gpt-4o-mini- Aliases:
gpt-4o-latest→gpt-4o
ClaudeProvider(
api_key: str,
base_url: str = "https://api.anthropic.com",
max_retries: int = 3,
timeout: int = 60
)Supported Models (2025 Pricing):
claude-sonnet-4,claude-sonnet-3.7,claude-sonnet-3.5- Aliases:
claude-4→claude-sonnet-4
GeminiProvider(
api_key: str,
base_url: str = "https://generativelanguage.googleapis.com",
max_retries: int = 3,
timeout: int = 60
)Supported Models (2025 Pricing):
gemini-2.5-pro,gemini-2.5-flash,gemini-2.5-flash-litegemini-2.0-flash,gemini-2.0-flash-lite- Aliases:
gemini-2.5→gemini-2.5-pro
Perfect for homework help platforms where you want to guide learning without giving away answers:
# Student asks: "What is the quadratic formula?"
# Instead of giving the formula directly, VETTING guides:
# "Great question! Let's think about this step by step.
# What do you know about quadratic equations? What form do they take?"Maintain assessment integrity while still providing help:
# During an exam, student asks for direct answer
# VETTING detects this violates policy and provides guidance instead:
# "I can't give you the direct answer, but I can help you think through
# the problem. What approach would you take to solve this type of question?"Prevent harmful or inappropriate responses while maintaining helpful interaction:
# User asks about dangerous activities
# VETTING detects safety concern and responds appropriately:
# "[REQUIRES_ATTENTION] I understand you're curious, but I can't provide
# information that could be harmful. Instead, let me suggest some safe
# alternatives..."Ensure training materials adhere to company policies and learning objectives:
# Training scenario with specific learning outcomes
# VETTING ensures responses align with corporate training goals
# while preventing disclosure of confidential information- Use environment variables for API keys and basic settings
- Use configuration files for complex setups
- Validate configurations before use
- Keep verification model parameters more conservative (lower temperature)
- Enable cost tracking in production
- Monitor usage patterns and optimize model selection
- Use cheaper models for verification when possible
- Set up cost alerts for production systems
- Design clear learning objectives for context items
- Use specific key concepts to avoid revealing
- Set appropriate maximum attempts (2-3 for homework, 1 for assessments)
- Always include explanations in answer keys for better verification
- Always validate inputs before processing
- Implement proper retry logic for provider failures
- Log verification failures for analysis
- Have fallback responses for system errors
- Use connection pooling for high-throughput applications
- Implement proper monitoring and alerting
- Cache provider instances to avoid recreation overhead
- Set up log aggregation for debugging
The vetting_python/examples/ directory contains comprehensive examples:
basic_usage.py- Getting started examplesadvanced_usage.py- Complex scenarios and custom providersintegration_patterns.py- Web API and platform integration examples
Run the examples:
cd vetting_python/examples
python basic_usage.py
python advanced_usage.py
python integration_patterns.py- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
git clone https://github.com/your-org/vetting-python.git
cd vetting-python
pip install -e ".[dev]"pytest tests/black vetting_python/
isort vetting_python/
mypy vetting_python/If you use VETTING in your research, please cite our paper (citation will be updated upon publication):
@misc{vetting2025,
title={VETTING: Verification and Evaluation Tool for Targeting Invalid Narrative Generation},
author={VETTING Research Team},
year={2025},
note={Available at: https://github.com/hichipli/vetting-python}
}This project is licensed under the MIT License - see the LICENSE file for details.
- 📚 Documentation: README.md
- 🐛 Issues: GitHub Issues
- 💬 Discussions: GitHub Discussions
- 🌐 Research Lab: VIABLE Lab
- 📧 Contact: Contact Form or hli3@ufl.edu
- ✅ Published to PyPI: vetting-python
- ✅ Installation:
pip install vetting-python - ✅ Dual-LLM architecture implementation
- ✅ OpenAI, Claude, and Gemini provider support
- ✅ Educational vetting capabilities
- ✅ Cost tracking and monitoring
- ✅ Comprehensive configuration system
- ✅ Safety feature integration
- ✅ Example applications and documentation
- ✅ Automated publishing workflow with GitHub Actions
- ✅ Security and contribution guidelines
Built with ❤️ for safer and more effective AI interactions in education and beyond.