Skip to content

CONFIGURATION_MANAGEMENT_IMPLEMENTATION_PLAN.md

Latest commit

 

History

History
1279 lines (987 loc) · 38.1 KB

CONFIGURATION_MANAGEMENT_IMPLEMENTATION_PLAN.md

File metadata and controls

1279 lines (987 loc) · 38.1 KB
Raw

Configuration Management Implementation Plan

Created: 2025-10-09 Status: Planning Phase Complete Target Completion: 2025-10-16

Executive Summary

This document provides a comprehensive implementation plan for adding 4 integrated configuration management features to AutoBot's backend:

  1. Environment Variable Priority Enforcement - Already partially implemented, needs enhancement
  2. Configuration Validation with Startup Warnings - New feature
  3. API Endpoint for GUI Settings Sync - New feature
  4. File Watching for Hot Reload - New feature with distributed support

Table of Contents

  1. Current State Analysis
  2. Feature 1: Environment Variable Enforcement
  3. Feature 2: Configuration Validation
  4. Feature 3: Settings Sync API
  5. Feature 4: Hot Reload System
  6. Integration Plan
  7. Testing Strategy
  8. Deployment Strategy
  9. Risk Assessment

Current State Analysis

Existing Infrastructure

File: src/unified_config_manager.py

What Works:

  • Environment variable override system with AUTOBOT_* prefix (line 260-317)
  • Configuration precedence: config.yaml → settings.json → environment variables
  • Comprehensive env var mappings for 30+ settings
  • Type conversion (bool, int, string)
  • Async file operations infrastructure (line 597-634)
  • Callback system placeholder (line 713-729)

What's Missing:

  • No validation warnings or conflict detection
  • No sync-to-base API endpoint
  • No file watching mechanism
  • No distributed configuration sync
  • No atomic file write operations
  • No startup validation integration

Integration Points

File: backend/app_factory.py

Startup Sequence:

# Line 471-487: Configuration loads at 10% progress
config = UnifiedConfigManager()
app.state.config = config
# OPPORTUNITY: Add validation here (after line 487)

Dependency Injection:

# backend/dependencies.py line 14-24
def get_config() -> UnifiedConfigManager:
    return global_config_manager

Feature 1: Environment Variable Enforcement

Status: ENHANCEMENT NEEDED

Current Implementation

Location: unified_config_manager.py line 260-317

def _apply_env_overrides(self) -> None:
    """Apply environment variable overrides using AUTOBOT_ prefix"""
    env_overrides = {}

    # Environment variable mappings
    env_mappings = {
        "AUTOBOT_BACKEND_HOST": ["backend", "server_host"],
        "AUTOBOT_BACKEND_PORT": ["backend", "server_port"],
        # ... 28 more mappings
    }

    for env_var, config_path in env_mappings.items():
        env_value = os.getenv(env_var)
        if env_value is not None:
            # Convert and apply

Enhancement Requirements

1.1 Add Logging for Applied Overrides

Current: Logs each override individually (line 313) Enhancement: Aggregate and log summary

def _apply_env_overrides(self) -> Dict[str, Any]:
    """Apply environment variable overrides and return summary"""
    env_overrides = {}
    applied_overrides = []

    # ... existing logic ...

    if env_overrides:
        self._config = self._deep_merge(self._config, env_overrides)
        logger.info(f"Applied {len(applied_overrides)} environment variable overrides")
        for override in applied_overrides:
            logger.debug(f"  {override['var']} = {override['value']}")

    return {"applied": applied_overrides, "total": len(env_mappings)}

1.2 Expand Env Var Coverage

Missing Mappings to Add:

# Security settings
"AUTOBOT_SESSION_TIMEOUT": ["security", "session_timeout_minutes"],
"AUTOBOT_ENABLE_ENCRYPTION": ["security", "enable_encryption"],

# Knowledge base
"AUTOBOT_KB_UPDATE_FREQUENCY": ["knowledge_base", "update_frequency_days"],
"AUTOBOT_KB_ENABLED": ["knowledge_base", "enabled"],

# Developer mode
"AUTOBOT_DEVELOPER_MODE": ["ui", "developer_mode"],
"AUTOBOT_DEBUG_LOGGING": ["developer", "debug_logging"],

# Voice interface
"AUTOBOT_VOICE_ENABLED": ["voice_interface", "enabled"],
"AUTOBOT_VOICE_RATE": ["voice_interface", "speech_rate"],

1.3 Implementation Steps

  1. Extract env var mapping to separate method (easier to test and extend)
  2. Add validation of env var values (before applying)
  3. Return override summary for integration with validation system
  4. Add env var documentation in CLAUDE.md

Estimated Effort: 2 hours Priority: Medium (enhancement of existing feature)

Feature 2: Configuration Validation

Status: NEW FEATURE - HIGH PRIORITY

Design Overview

Purpose: Detect configuration conflicts and invalid values during startup and reload operations.

Architecture

2.1 Validation Levels

class ValidationLevel(Enum):
    INFO = "info"          # Expected behavior (env var overrides file)
    WARNING = "warning"    # Potential issue (conflicting values)
    ERROR = "error"        # Invalid configuration (wrong type/range)
    CRITICAL = "critical"  # Service will fail (missing required value)

2.2 Validation Checks

Structure Validation:

def _validate_structure(self) -> List[ValidationResult]:
    """Ensure required configuration sections exist"""
    required_sections = ["backend", "memory", "ui", "chat", "logging"]
    issues = []

    for section in required_sections:
        if section not in self._config:
            issues.append(ValidationResult(
                level=ValidationLevel.CRITICAL,
                category="structure",
                message=f"Missing required section: {section}",
                path=section
            ))

    return issues

Type Validation:

def _validate_types(self) -> List[ValidationResult]:
    """Validate configuration value types"""
    type_specs = {
        "backend.server_port": int,
        "backend.timeout": int,
        "backend.streaming": bool,
        "memory.redis.enabled": bool,
        "memory.redis.port": int,
        # ... more specs
    }

    issues = []
    for path, expected_type in type_specs.items():
        value = self.get_nested(path)
        if value is not None and not isinstance(value, expected_type):
            issues.append(ValidationResult(
                level=ValidationLevel.ERROR,
                category="type",
                message=f"Expected {expected_type.__name__}, got {type(value).__name__}",
                path=path,
                value=value
            ))

    return issues

Range Validation:

def _validate_ranges(self) -> List[ValidationResult]:
    """Validate numeric ranges"""
    range_specs = {
        "backend.server_port": (1, 65535),
        "backend.timeout": (1, 300),
        "memory.redis.port": (1, 65535),
        "chat.max_messages": (10, 10000),
        "security.session_timeout_minutes": (5, 1440),
    }

    issues = []
    for path, (min_val, max_val) in range_specs.items():
        value = self.get_nested(path)
        if value is not None and not (min_val <= value <= max_val):
            issues.append(ValidationResult(
                level=ValidationLevel.ERROR,
                category="range",
                message=f"Value {value} outside valid range [{min_val}, {max_val}]",
                path=path,
                value=value
            ))

    return issues

Conflict Detection:

def _detect_conflicts(self) -> List[ValidationResult]:
    """Detect conflicts between configuration sources"""
    issues = []

    # Load raw sources
    base_config = self._load_yaml_config()
    user_settings = self._load_json_settings()

    # Check for complete section replacements
    for section in ["backend.cors_origins", "backend.llm.ollama.models"]:
        base_value = self._get_nested_from_dict(base_config, section)
        user_value = self._get_nested_from_dict(user_settings, section)

        if base_value and user_value and len(base_value) != len(user_value):
            issues.append(ValidationResult(
                level=ValidationLevel.WARNING,
                category="conflict",
                message=f"settings.json completely replaces {section} from config.yaml "
                        f"({len(base_value)} items → {len(user_value)} items)",
                path=section
            ))

    # Check for env var conflicts
    env_overrides = self._get_applied_env_overrides()
    for env_var, path in env_overrides.items():
        file_value = self._get_nested_from_dict(user_settings or base_config, path)
        env_value = os.getenv(env_var)

        if file_value != env_value:
            issues.append(ValidationResult(
                level=ValidationLevel.INFO,
                category="override",
                message=f"Environment variable {env_var} overrides file value",
                path=".".join(path),
                details={"file": file_value, "env": env_value}
            ))

    return issues

2.3 Main Validation Method

def _validate_configuration_consistency(self) -> ValidationReport:
    """
    Comprehensive configuration validation.

    Returns:
        ValidationReport with all detected issues organized by level
    """
    all_issues = []

    # Run all validation checks
    all_issues.extend(self._validate_structure())
    all_issues.extend(self._validate_types())
    all_issues.extend(self._validate_ranges())
    all_issues.extend(self._validate_url_formats())
    all_issues.extend(self._validate_file_paths())
    all_issues.extend(self._detect_conflicts())

    # Organize by level
    report = ValidationReport(
        timestamp=datetime.now(timezone.utc),
        total_issues=len(all_issues),
        critical=len([i for i in all_issues if i.level == ValidationLevel.CRITICAL]),
        errors=len([i for i in all_issues if i.level == ValidationLevel.ERROR]),
        warnings=len([i for i in all_issues if i.level == ValidationLevel.WARNING]),
        info=len([i for i in all_issues if i.level == ValidationLevel.INFO]),
        issues=all_issues
    )

    return report

2.4 Integration with Startup

File: backend/app_factory.py line 482-487

# BEFORE (current)
logger.info("✅ [ 10%] Config: Loading unified configuration...")
config = UnifiedConfigManager()
app.state.config = config
app_state["config"] = config
logger.info("✅ [ 10%] Config: Configuration loaded successfully")

# AFTER (with validation)
logger.info("✅ [ 10%] Config: Loading unified configuration...")
config = UnifiedConfigManager()
app.state.config = config
app_state["config"] = config
logger.info("✅ [ 10%] Config: Configuration loaded successfully")

# NEW: Validate configuration
logger.info("🔍 [ 11%] Config: Validating configuration consistency...")
validation_report = config.validate_configuration_consistency()
app.state.config_validation = validation_report

if validation_report.critical > 0:
    logger.error(f"❌ CRITICAL: {validation_report.critical} critical configuration issues detected")
    for issue in validation_report.get_critical():
        logger.error(f"  - {issue.message} (path: {issue.path})")
    raise RuntimeError("Critical configuration issues prevent startup")

if validation_report.errors > 0:
    logger.warning(f"⚠️ {validation_report.errors} configuration errors detected (non-blocking)")
    for issue in validation_report.get_errors():
        logger.warning(f"  - {issue.message} (path: {issue.path})")

if validation_report.warnings > 0:
    logger.info(f"ℹ️ {validation_report.warnings} configuration warnings (informational)")
    for issue in validation_report.get_warnings():
        logger.debug(f"  - {issue.message} (path: {issue.path})")

logger.info(f"✅ [ 12%] Config: Validation complete ({validation_report.total_issues} items)")

2.5 Validation API Endpoint

New File: autobot-backend/api/config_validation.py

from fastapi import APIRouter, Depends
from backend.dependencies import get_config
from src.unified_config_manager import UnifiedConfigManager

router = APIRouter()

@router.get("/api/config/validate")
async def validate_configuration(config: UnifiedConfigManager = Depends(get_config)):
    """
    Validate current configuration and return detailed report.

    Returns:
        ValidationReport with all detected issues
    """
    report = config.validate_configuration_consistency()

    return {
        "status": "healthy" if report.critical == 0 else "unhealthy",
        "timestamp": report.timestamp.isoformat(),
        "summary": {
            "total": report.total_issues,
            "critical": report.critical,
            "errors": report.errors,
            "warnings": report.warnings,
            "info": report.info
        },
        "issues": [
            {
                "level": issue.level.value,
                "category": issue.category,
                "message": issue.message,
                "path": issue.path,
                "value": issue.value,
                "details": issue.details
            }
            for issue in report.issues
        ]
    }

2.6 Implementation Steps

Phase 2.1: Core Validation (4 hours)

  1. Create ValidationLevel enum
  2. Create ValidationResult and ValidationReport data classes
  3. Implement _validate_structure(), _validate_types(), _validate_ranges()
  4. Add _validate_configuration_consistency() method
  5. Unit tests for validation logic

Phase 2.2: Conflict Detection (3 hours)

  1. Implement _detect_conflicts() method
  2. Add raw source loading helpers
  3. Test conflict detection scenarios
  4. Document expected vs unexpected conflicts

Phase 2.3: Startup Integration (2 hours)

  1. Modify app_factory.py startup sequence
  2. Add validation report to app state
  3. Add critical issue blocking logic
  4. Test startup with various config states

Phase 2.4: API Endpoint (2 hours)

  1. Create autobot-backend/api/config_validation.py
  2. Add router to app_factory.py
  3. Add frontend integration docs
  4. Test API endpoint

Total Estimated Effort: 11 hours Priority: HIGH (critical for troubleshooting)

Feature 3: Settings Sync API

Status: NEW FEATURE - MEDIUM PRIORITY

Design Overview

Purpose: Allow GUI to persist changes from settings.json back to config.yaml as the new baseline.

Endpoint: POST /api/settings/sync-to-base

Architecture

3.1 Atomic Write Implementation

import fcntl
import tempfile
from pathlib import Path
from contextlib import contextmanager

@contextmanager
def file_lock(file_path: Path, timeout: float = 5.0):
    """
    Context manager for file locking with timeout.

    Prevents race conditions during concurrent config file access.
    """
    lock_path = file_path.with_suffix(file_path.suffix + '.lock')
    lock_file = None

    try:
        lock_file = open(lock_path, 'w')

        # Try to acquire exclusive lock with timeout
        start_time = time.time()
        while True:
            try:
                fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
                break
            except BlockingIOError:
                if time.time() - start_time > timeout:
                    raise TimeoutError(f"Could not acquire lock on {file_path} after {timeout}s")
                time.sleep(0.1)

        yield lock_file

    finally:
        if lock_file:
            try:
                fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN)
                lock_file.close()
                lock_path.unlink(missing_ok=True)
            except Exception as e:
                logger.warning(f"Error releasing lock: {e}")


def _atomic_write_yaml(self, file_path: Path, data: Dict[str, Any]) -> None:
    """
    Write configuration to YAML file atomically.

    Uses temp file + rename pattern to ensure atomic write operation.
    """
    # Create temp file in same directory (required for atomic rename)
    temp_file = file_path.with_suffix('.tmp')

    try:
        # Write to temp file
        with open(temp_file, 'w', encoding='utf-8') as f:
            yaml.dump(data, f, default_flow_style=False, indent=2)

        # Validate temp file is valid YAML
        with open(temp_file, 'r', encoding='utf-8') as f:
            yaml.safe_load(f)

        # Atomic rename (replaces destination if exists)
        temp_file.rename(file_path)

        logger.info(f"Successfully wrote configuration to {file_path}")

    except Exception as e:
        # Clean up temp file on error
        temp_file.unlink(missing_ok=True)
        raise RuntimeError(f"Failed to write configuration: {e}")

3.2 Sync-to-Base Method

def sync_settings_to_base(self) -> Dict[str, Any]:
    """
    Sync current settings.json to config.yaml as new baseline.

    This makes user customizations the new default configuration.

    Returns:
        Dict with operation status and details
    """
    with file_lock(self.base_config_file, timeout=5.0):
        try:
            # Load current merged configuration
            current_config = self._config.copy()

            # Filter out prompts (not stored in config.yaml)
            if "prompts" in current_config:
                del current_config["prompts"]

            # Validate before writing
            validation_report = self._validate_configuration_consistency()
            if validation_report.critical > 0:
                return {
                    "status": "error",
                    "message": "Cannot sync: critical configuration issues detected",
                    "validation": validation_report.to_dict()
                }

            # Backup current config.yaml
            backup_path = self.base_config_file.with_suffix('.yaml.backup')
            if self.base_config_file.exists():
                import shutil
                shutil.copy2(self.base_config_file, backup_path)

            # Write atomically
            self._atomic_write_yaml(self.base_config_file, current_config)

            # Clear cache to force reload
            self._sync_cache_timestamp = None

            # Remove settings.json (now redundant)
            if self.settings_file.exists():
                self.settings_file.unlink()
                logger.info("Removed settings.json (now baseline)")

            return {
                "status": "success",
                "message": "Settings synced to base configuration",
                "backup_created": str(backup_path),
                "validation": validation_report.summary()
            }

        except Exception as e:
            logger.error(f"Sync-to-base failed: {e}")
            return {
                "status": "error",
                "message": str(e)
            }

3.3 API Endpoint with Rate Limiting

File: autobot-backend/api/settings_sync.py

from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi.responses import JSONResponse
from backend.dependencies import get_config
from src.unified_config_manager import UnifiedConfigManager
import time
from collections import defaultdict
from datetime import datetime, timedelta

router = APIRouter()

# Simple in-memory rate limiter
rate_limit_store = defaultdict(list)
RATE_LIMIT_REQUESTS = 1
RATE_LIMIT_WINDOW = 60  # seconds

def check_rate_limit(request: Request) -> bool:
    """Check if request is within rate limit"""
    client_ip = request.client.host
    now = time.time()

    # Clean old requests
    rate_limit_store[client_ip] = [
        req_time for req_time in rate_limit_store[client_ip]
        if now - req_time < RATE_LIMIT_WINDOW
    ]

    # Check limit
    if len(rate_limit_store[client_ip]) >= RATE_LIMIT_REQUESTS:
        return False

    # Add current request
    rate_limit_store[client_ip].append(now)
    return True


@router.post("/api/settings/sync-to-base")
async def sync_settings_to_base(
    request: Request,
    config: UnifiedConfigManager = Depends(get_config)
):
    """
    Sync current settings.json to config.yaml as new baseline.

    Rate limited to 1 request per minute to prevent accidental overwrites.

    Returns:
        Operation status with backup file path
    """
    # Rate limiting
    if not check_rate_limit(request):
        raise HTTPException(
            status_code=429,
            detail="Rate limit exceeded. Only 1 sync operation per minute allowed."
        )

    # Perform sync
    result = config.sync_settings_to_base()

    if result["status"] == "error":
        raise HTTPException(status_code=500, detail=result["message"])

    return JSONResponse(content=result)


@router.get("/api/settings/sync-status")
async def get_sync_status(config: UnifiedConfigManager = Depends(get_config)):
    """
    Get information about sync-to-base operation readiness.

    Returns:
        Status information including if settings.json exists and validation state
    """
    settings_exists = config.settings_file.exists()
    validation_report = config.validate_configuration_consistency()

    return {
        "settings_file_exists": settings_exists,
        "can_sync": settings_exists and validation_report.critical == 0,
        "validation_summary": validation_report.summary(),
        "backup_exists": config.base_config_file.with_suffix('.yaml.backup').exists()
    }

3.4 Implementation Steps

Phase 3.1: Atomic Write (3 hours)

  1. Implement file_lock() context manager
  2. Implement _atomic_write_yaml() method
  3. Add error handling and cleanup
  4. Unit tests for atomic operations

Phase 3.2: Sync Method (3 hours)

  1. Implement sync_settings_to_base() method
  2. Add backup creation logic
  3. Add validation integration
  4. Add settings.json cleanup
  5. Unit tests for sync logic

Phase 3.3: API Endpoint (2 hours)

  1. Create autobot-backend/api/settings_sync.py
  2. Implement rate limiting
  3. Add sync status endpoint
  4. Add router to app_factory.py

Phase 3.4: Documentation (1 hour)

  1. Document API endpoints
  2. Add usage examples
  3. Document backup/restore procedure

Total Estimated Effort: 9 hours Priority: MEDIUM (useful for GUI integration)

Feature 4: Hot Reload System

Status: NEW FEATURE - HIGH COMPLEXITY

Design Overview

Purpose: Automatically detect configuration file changes and reload across all VMs in distributed architecture.

Architecture

4.1 File Watching with Watchdog

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import asyncio

class ConfigFileHandler(FileSystemEventHandler):
    """
    File system event handler for configuration files.

    Implements debouncing to handle rapid successive changes.
    """

    def __init__(self, config_manager, debounce_seconds: float = 0.5):
        self.config_manager = config_manager
        self.debounce_seconds = debounce_seconds
        self._pending_reload = None
        self._reload_lock = asyncio.Lock()

    def on_modified(self, event):
        """Handle file modification events"""
        if event.is_directory:
            return

        # Only watch config files
        if event.src_path not in [
            str(self.config_manager.base_config_file),
            str(self.config_manager.settings_file)
        ]:
            return

        logger.info(f"Configuration file modified: {event.src_path}")

        # Cancel pending reload
        if self._pending_reload:
            self._pending_reload.cancel()

        # Schedule debounced reload
        loop = asyncio.get_event_loop()
        self._pending_reload = loop.call_later(
            self.debounce_seconds,
            lambda: asyncio.create_task(self._perform_reload(event.src_path))
        )

    async def _perform_reload(self, file_path: str):
        """
        Perform the actual configuration reload.

        Acquires lock to prevent concurrent reloads.
        """
        async with self._reload_lock:
            try:
                logger.info(f"Reloading configuration from {file_path}...")

                # Reload configuration
                self.config_manager.reload()

                # Validate new configuration
                validation_report = self.config_manager.validate_configuration_consistency()

                if validation_report.critical > 0:
                    logger.error("Configuration reload resulted in critical issues")
                    for issue in validation_report.get_critical():
                        logger.error(f"  - {issue.message}")
                    # Reload from backup?
                    return

                # Notify callbacks
                await self.config_manager._notify_callbacks("main", self.config_manager.to_dict())

                # Publish reload notification to Redis (for other VMs)
                await self._publish_reload_notification(file_path)

                logger.info("Configuration reloaded successfully")

            except Exception as e:
                logger.error(f"Configuration reload failed: {e}")

    async def _publish_reload_notification(self, file_path: str):
        """
        Publish reload notification to Redis pub/sub.

        Other VMs subscribe to this channel and reload their local configs.
        """
        try:
            from src.utils.async_redis_manager import AsyncRedisManager
            redis_manager = AsyncRedisManager()

            await redis_manager.publish_message(
                channel="config:reload",
                message={
                    "timestamp": datetime.now(timezone.utc).isoformat(),
                    "source_vm": os.getenv("VM_HOSTNAME", "unknown"),
                    "file_path": file_path,
                    "action": "reload"
                }
            )

            logger.info("Published reload notification to Redis")

        except Exception as e:
            logger.warning(f"Failed to publish reload notification: {e}")

4.2 File Watcher Management

async def start_file_watcher(self) -> None:
    """
    Start watching configuration files for changes.

    Initializes watchdog observer and Redis pub/sub subscriber.
    """
    if hasattr(self, '_file_watcher_observer'):
        logger.warning("File watcher already running")
        return

    try:
        # Start watchdog observer
        event_handler = ConfigFileHandler(self)
        observer = Observer()
        observer.schedule(
            event_handler,
            str(self.config_dir),
            recursive=False
        )
        observer.start()

        self._file_watcher_observer = observer
        self._file_watcher_handler = event_handler

        logger.info(f"File watcher started for {self.config_dir}")

        # Start Redis subscriber (for distributed reload)
        await self._start_redis_subscriber()

    except Exception as e:
        logger.error(f"Failed to start file watcher: {e}")
        raise


async def stop_file_watcher(self) -> None:
    """Stop file watching and cleanup resources"""
    if hasattr(self, '_file_watcher_observer'):
        self._file_watcher_observer.stop()
        self._file_watcher_observer.join(timeout=5.0)
        delattr(self, '_file_watcher_observer')
        logger.info("File watcher stopped")

    await self._stop_redis_subscriber()

4.3 Distributed Reload via Redis Pub/Sub

async def _start_redis_subscriber(self) -> None:
    """
    Start Redis pub/sub subscriber for distributed reload notifications.

    Listens on 'config:reload' channel for reload messages from other VMs.
    """
    try:
        from src.utils.async_redis_manager import AsyncRedisManager

        self._redis_manager = AsyncRedisManager()

        async def reload_handler(message: Dict[str, Any]):
            """Handle reload notification from other VMs"""
            source_vm = message.get("source_vm")
            current_vm = os.getenv("VM_HOSTNAME", "unknown")

            # Ignore our own messages
            if source_vm == current_vm:
                return

            logger.info(f"Received reload notification from {source_vm}")

            # Reload local configuration
            try:
                self.reload()
                await self._notify_callbacks("main", self.to_dict())
                logger.info("Configuration reloaded from remote notification")
            except Exception as e:
                logger.error(f"Failed to reload from notification: {e}")

        # Subscribe to reload channel
        self._redis_subscriber_task = asyncio.create_task(
            self._redis_manager.subscribe_channel("config:reload", reload_handler)
        )

        logger.info("Redis subscriber started for distributed config reload")

    except Exception as e:
        logger.warning(f"Failed to start Redis subscriber (graceful degradation): {e}")


async def _stop_redis_subscriber(self) -> None:
    """Stop Redis pub/sub subscriber"""
    if hasattr(self, '_redis_subscriber_task'):
        self._redis_subscriber_task.cancel()
        try:
            await self._redis_subscriber_task
        except asyncio.CancelledError:
            pass

        delattr(self, '_redis_subscriber_task')
        logger.info("Redis subscriber stopped")

4.4 Integration with App Startup

File: backend/app_factory.py - add to background initialization (line 547-616)

# In background_init() function, after Memory Graph initialization (~line 586)

# Initialize Configuration Hot Reload - NON-CRITICAL
logger.info("✅ [ 95%] Config Hot Reload: Initializing file watcher...")
try:
    await app.state.config.start_file_watcher()
    logger.info("✅ [ 95%] Config Hot Reload: File watcher active")
except Exception as reload_error:
    logger.warning(f"Config hot reload initialization failed: {reload_error}")
    logger.warning("Configuration will require manual reload")

logger.info("✅ [100%] PHASE 2 COMPLETE: All background services initialized")

4.5 Graceful Degradation

When Redis is unavailable:

async def _publish_reload_notification(self, file_path: str):
    """Publish with graceful fallback"""
    try:
        # Try Redis pub/sub
        await self._redis_publish(...)
    except Exception as e:
        logger.warning(f"Redis unavailable, reload is local-only: {e}")
        # Continue without distributed reload

When watchdog is unavailable:

async def start_file_watcher(self) -> None:
    """Start with fallback to polling"""
    try:
        # Try watchdog
        observer = Observer()
        ...
    except ImportError:
        logger.warning("watchdog not available, using polling fallback")
        await self._start_polling_watcher()

4.6 Implementation Steps

Phase 4.1: Local File Watching (4 hours)

  1. Add watchdog dependency
  2. Implement ConfigFileHandler class
  3. Implement debouncing logic
  4. Add start_file_watcher() and stop_file_watcher() methods
  5. Unit tests for file watching

Phase 4.2: Reload Logic (3 hours)

  1. Implement _perform_reload() method
  2. Add validation integration
  3. Add callback notification
  4. Add error handling and rollback
  5. Test reload scenarios

Phase 4.3: Distributed Sync (4 hours)

  1. Implement Redis pub/sub publisher
  2. Implement Redis pub/sub subscriber
  3. Add VM identification logic
  4. Add message filtering (ignore own messages)
  5. Test cross-VM reload

Phase 4.4: Graceful Degradation (2 hours)

  1. Add fallback for Redis unavailable
  2. Add fallback for watchdog unavailable
  3. Add polling implementation
  4. Test degraded modes

Phase 4.5: Integration (2 hours)

  1. Add to app_factory.py background init
  2. Add cleanup to shutdown sequence
  3. Integration testing
  4. Documentation

Total Estimated Effort: 15 hours Priority: HIGH (significant operational improvement)

Integration Plan

Phase 1: Environment Variable Enhancement (Week 1, Day 1-2)

  • Expand env var coverage
  • Improve logging
  • Document new variables
  • Deliverable: Enhanced environment variable system

Phase 2: Configuration Validation (Week 1, Day 3-5)

  • Implement validation framework
  • Add startup integration
  • Create API endpoint
  • Deliverable: Validation system with startup warnings

Phase 3: Settings Sync API (Week 2, Day 1-2)

  • Implement atomic writes
  • Create sync-to-base endpoint
  • Add rate limiting
  • Deliverable: Settings sync API

Phase 4: Hot Reload System (Week 2, Day 3-5)

  • Implement file watching
  • Add distributed reload
  • Integrate with startup
  • Deliverable: Hot reload with distributed support

Phase 5: Integration Testing (Week 3, Day 1-2)

  • Test all features together
  • Verify distributed behavior
  • Performance testing
  • Deliverable: Fully integrated system

Phase 6: Documentation & Deployment (Week 3, Day 3)

  • Update CLAUDE.md
  • Create API documentation
  • Deployment guide
  • Deliverable: Production-ready feature set

Testing Strategy

Unit Tests

File: tests/unit/test_config_validation.py

def test_validate_structure():
    """Test structure validation detects missing sections"""

def test_validate_types():
    """Test type validation catches type mismatches"""

def test_validate_ranges():
    """Test range validation enforces limits"""

def test_detect_conflicts():
    """Test conflict detection identifies source conflicts"""

File: tests/unit/test_settings_sync.py

def test_atomic_write():
    """Test atomic write creates temp file and renames"""

def test_file_locking():
    """Test file locking prevents concurrent writes"""

def test_sync_to_base():
    """Test sync-to-base copies settings to config"""

File: tests/unit/test_hot_reload.py

def test_file_watcher():
    """Test file watcher detects changes"""

def test_debouncing():
    """Test debouncing batches rapid changes"""

def test_reload_notification():
    """Test Redis pub/sub notification"""

Integration Tests

File: tests/integration/test_config_management.py

async def test_startup_validation():
    """Test validation runs during app startup"""

async def test_config_reload():
    """Test configuration reload updates app state"""

async def test_distributed_reload():
    """Test reload propagates across VMs"""

Performance Tests

File: tests/performance/test_config_performance.py

def test_validation_performance():
    """Ensure validation completes in < 100ms"""

def test_reload_performance():
    """Ensure reload completes in < 500ms"""

def test_concurrent_access():
    """Test file locking under concurrent access"""

Deployment Strategy

Dependencies

Add to requirements.txt:

watchdog>=3.0.0  # File system event monitoring

Environment Variables

Document in CLAUDE.md:

# Configuration Management
AUTOBOT_CONFIG_AUTO_RELOAD=true          # Enable hot reload
AUTOBOT_CONFIG_VALIDATION_ON_STARTUP=true # Validate on startup
AUTOBOT_CONFIG_DISTRIBUTED_RELOAD=true    # Use Redis for distributed reload
VM_HOSTNAME=vm1-frontend                  # VM identifier for distributed reload

Deployment Checklist

  1. Install Dependencies:

    pip install -r requirements.txt

  2. Update Configuration:

    • No breaking changes to existing config files
    • New features opt-in via environment variables

  3. Deploy to VMs:

    # Use sync-to-vm script to deploy updated code
./scripts/utilities/sync-to-vm.sh all src/ /home/autobot/src/
./scripts/utilities/sync-to-vm.sh all backend/ /home/autobot/backend/

  4. Verify Deployment:

    # Check validation endpoint
curl https://<backend-ip>:8443/api/config/validate

# Check sync status
curl https://<backend-ip>:8443/api/settings/sync-status

  5. Enable Hot Reload:

    # Set environment variable on all VMs
export AUTOBOT_CONFIG_AUTO_RELOAD=true

# Restart backend
sudo systemctl restart autobot-backend

Risk Assessment

High Risks

Risk: Configuration file corruption during atomic write Mitigation: Backup before write, validate before rename, rollback on error Impact: Service disruption

Risk: Redis unavailable during distributed reload Mitigation: Graceful degradation to local-only reload Impact: Reduced functionality

Risk: File locking timeout on slow filesystem Mitigation: Configurable timeout, async operations Impact: Failed sync operations

Medium Risks

Risk: Validation false positives blocking startup Mitigation: Separate critical vs warning levels, non-blocking warnings Impact: Delayed startup

Risk: Rate limiting too restrictive Mitigation: Configurable rate limits, clear error messages Impact: User frustration

Low Risks

Risk: File watcher memory leak Mitigation: Proper cleanup in shutdown, watchdog validated Impact: Memory consumption

Risk: Debouncing delay too long Mitigation: Configurable debounce time (default 500ms) Impact: Slow reload response

Success Criteria

Feature 1: Environment Variables

  • All 40+ configuration paths support env var override
  • Override summary logged on startup
  • Documentation updated in CLAUDE.md

Feature 2: Validation

  • Validation completes in < 100ms
  • Critical issues block startup
  • Warnings logged but non-blocking
  • API endpoint returns detailed report

Feature 3: Sync API

  • Atomic writes prevent corruption
  • File locking prevents race conditions
  • Rate limiting prevents accidents
  • Backup created before sync

Feature 4: Hot Reload

  • File changes detected in < 1 second
  • Reload completes in < 500ms
  • Distributed reload propagates to all VMs
  • Graceful degradation when Redis unavailable

Overall

  • No breaking changes to existing deployments
  • All tests passing (unit, integration, performance)
  • Documentation complete
  • Deployed to all 6 machines (main + 5 VMs)

Conclusion

This implementation plan provides a comprehensive approach to adding advanced configuration management features to AutoBot's backend. The phased approach ensures incremental delivery of value while managing complexity.

Estimated Total Effort: 37 hours (approximately 1 week with proper focus)

Key Benefits:

  1. Better troubleshooting with validation warnings
  2. Reduced configuration errors via conflict detection
  3. Improved GUI integration with sync-to-base API
  4. Operational excellence with hot reload

Next Steps:

  1. Review and approve this plan
  2. Begin Phase 1 implementation (Environment Variables)
  3. Iterate through phases with testing at each step
  4. Deploy and verify in production environment