feat(optimization): layer-by-layer inference mode for batch/offline processing #1946

Summary

Add a "memory-constrained" inference mode that uses airllm's layer-by-layer technique to run models exceeding available VRAM, targeting non-interactive batch workloads like knowledge base processing and research workflows.

Background

airllm enables running 70B–405B parameter models on as little as 4GB VRAM by loading one transformer layer at a time from disk, processing it, then evicting it. The tradeoff is speed — inference takes 30–120 seconds per response for 70B models.

This is not suitable for interactive chat but is viable for:

• AutoResearch background processing (Feature: AutoResearch integration — self-improving experiment loop with web search #1440)
• Knowledge base bulk analysis
• Code analysis batch jobs
• Offline evaluation and benchmarking
• Any batch task where quality matters more than latency

Currently AutoBot's largest local model is 13B. Layer-by-layer inference would allow running 70B+ models on the RTX 4070 (8GB VRAM) for tasks where the quality improvement justifies the latency cost.

Implementation

Approach A: Integrate airllm as a Provider

# New provider: llm_providers/airllm_provider.py
from airllm import AutoModel

class AirLLMProvider:
    def __init__(self, model_name: str):
        self.model = AutoModel.from_pretrained(
            model_name,
            compression="4bit",  # block-wise quantization
            profiling_mode=True,
        )
    
    async def generate(self, prompt: str, **kwargs) -> str:
        # Layer-by-layer inference — slow but fits in VRAM
        ...

Approach B: Implement the Pattern Natively

Implement the core technique (model splitting + sequential layer loading) in the NPU inference server as an alternative backend mode, without depending on airllm directly.

Prerequisites

1. Model splitting pipeline: one-time preprocessing to split models into per-layer safetensors files
2. Prefetch infrastructure (see related NPU prefetch issue)
3. KV cache management across sequential layer passes

Key Files

• autobot-backend/llm_interface_pkg/providers/ — new provider or mode
• autobot-infrastructure/shared/scripts/ai-ml/optimize_llm_models.py — model splitting step
• autobot-backend/llm_interface_pkg/optimization/router.py — routing to memory-constrained mode

Acceptance Criteria

• Layer-by-layer inference mode implemented (Approach A or B)
• Model splitting pipeline for one-time preprocessing
• Integration with OptimizationRouter — auto-selects when model exceeds VRAM
• Block-wise 4-bit quantization applied to layer files for faster loading
• Prefetch next layer while current executes
• Works for batch/offline tasks (not exposed to interactive chat by default)
• Benchmarks: latency per token, memory usage, accuracy vs full-precision

Effort

High effort, medium impact (enables new capabilities but limited to batch use cases).