[fantomx42]

Summary

Adds a comprehensive end-to-end benchmark harness (wheeler-baseline) that measures the performance delta of the full three-grid Wheeler architecture against a MiniLM nearest-neighbor baseline on context-conditioned recall tasks. This addresses a critical gap: while internal validation and SimLex/MMLU signals exist, the value of the full Wheeler system over raw MiniLM cosine similarity has never been measured head-to-head on the architecture's core claim (reconstruction-not-retrieval under context conditioning).

Key Changes

• New benchmark script (scripts/bench/baseline_harness.py, 863 lines):

  • Phase 1 — Topology signal isolation: Holds Pearson candidate set frozen and re-scores under three SCM topologies (zero/random/warmed) to measure whether SCM topology alone shifts top-1 results. Low flip rate indicates SCM carries little actionable signal at the recall layer.
  • Phase 2 — Head-to-head arms: Three competing systems scored on identical context-conditioned probes:
    • Arm 1 (MiniLM-cos): Raw cosine NN on MiniLM embeddings—the fair floor
    • Arm 2 (Wheeler-recognize): CA-frame Pearson without experiential/SCM reranking—isolates CA contribution
    • Arm 3 (Wheeler-full): Full three-grid with experiential priming and SCM-warmed interference scoring—the contender
  • Decision metric: Δ_recall@1 and Δ_MRR (Wheeler-full minus max of floor arms) with thresholds:
    • ≥ +0.10 → DYNAMICS EARN KEEP
    • ∈ (−0.05, 0.10) → INCONCLUSIVE
    • ≤ −0.05 → DYNAMICS ARE OVERHEAD (flag for KISS audit)
  • Output: JSONL with per-probe ranks/scores/keys plus TSV summary row for cross-run comparison

• Probe construction (_build_probes):

  • Selects passage pairs within cosine band [0.25, 0.55] to ensure overlap without near-duplication
  • Extracts shared cues (longest common content-token run) as ambiguous queries
  • Generates two probes per pair (one per context) with flipped correct answers

• Helper functions (inlined from scm_ab_eval.py):

  • _q_part: Extracts question portion (before " A:")
  • _paraphrase: Shuffles content tokens (drops stopwords) for query ambiguity
  • _content_tokens: Normalizes punctuation and filters stopwords
  • _find_rank: 1-indexed rank lookup
  • _ArmStats: Accumulates recall@1, recall@3, MRR per arm

• Experiential lifecycle:

  • _store_experiential_for_text: Evolves context under CA params, writes npy, returns hex_key for cleanup
  • _cleanup_experiential: Deletes experiential npy between probes to prevent residue bleed

• SCM warmup (mirrors scm_ab_eval):

  • Runs exact queries through recognize_top_k, loads top candidate attractors, computes interference scores
  • Updates SCMGrid.kappa_base via update_from_recall for warmed topology

• CLI entry point (wheeler-baseline):

  • Flags: --n (passages, default 50), --seed (default 42), --encoder (default "embedding"), --top-k (default 15), --datasets-dir, --no-warmup, --json-only
  • Requires arc.jsonl in datasets directory; exits with code 2 if missing
  • Requires sentence-transformers for MiniLM embedding; exits with code 2 if unavailable

• Test suite (tests/test_baseline_harness.py, 186 lines):

https://claude.ai/code/session_01AsEyLn5MaB3guTbXuTMGqB