bench(longhorizon): long-horizon context-management harness (10K→1M, walking skeleton)

[arybach]

Long-horizon context-management harness (10K → 1M tokens)

The measurement infra the enterprise Context-Runtime optimizations (F4 sparse retrieval, F2 adaptive materialization, F3 pattern memory) will be judged on. Each optimization is an arm, measured on the quality × cost × latency frontier against the frozen AGPL baseline, as available context grows 10K→1M tokens. AGPL, in redevops-rag/benchmarks (same Store/Embedder/hybrid_search stack as the cube/matrix); the enterprise overlay injects as a treatment arm and is not required to run.

Two design constraints (from the program owner)

1. Wikimedia corpus, two independent datasets — strategywiki (sha256-pinned, on disk) for dev; a different wiki (e.g. Simple English Wikipedia) as an independent holdout, so the harness isn't tuned to the data it's judged on.
2. The answerer must not manage context itself — DeepSeek-V4-Flash's built-in context management would confound our CR management with its internal one. The answerer is a plain fixed-window consumer (Qwen 32k / Kimi / grok). This is what makes the frontier attributable to the arm, and it sharpens the thesis: a small fixed-window answerer + CR context management, past the point the corpus exceeds the window, is the only way to answer — "better answers from less context → smaller self-hosted models made viable" at 1M scale.

Task

Real Wikimedia haystack + synthetic fact-needles at controlled depths → deterministic substring scoring (no LLM judge, so it's cheap and reproducible at 1M) with exact length control and no prior-gameability.

Arms

arm	feeds the fixed-window answerer	role
full	whole haystack, truncated to the window	the "give it everything" baseline to beat
cr	top-k retrieved (hybrid_search, scoped by document_ids)	AGPL context management
cr-enterprise	same, enterprise overlay injected	added when F4/F2/F3 ship

Proven now (dry smoke, no LLM/GPU)

--dry exercises the whole pipeline (corpus → needle insertion → embed → retrieve → score) with an oracle answerer. On strategywiki, window 32k, it already shows the frontier:

horizon arm     acc   in_tok
  10000 full    1.0    10305     # fits window: both correct, but full costs 2.5x the tokens
  10000 cr      1.0     4155
 200000 full    0.0    32000     # past the window: full drops every needle
 200000 cr      0.5-1.0 ~4-8k    # cr holds accuracy at flat token cost (limit-sensitive recall — what F4 targets)

Files

DESIGN.md, corpus.py (reuses the canonical MediaWiki parser), tasks.py (needle generator), arms.py (strategies + fixed-window answerer), run_longhorizon.py (runner + --dry).

Next (not in this PR)

Real fixed-window answerer runs (Qwen/Kimi) at the full 1M horizon; the independent holdout corpus; and the enterprise cr-enterprise arm once F4 lands. Generated result cells are gitignored (frozen deliberately as a control, like matrix_control/).

🤖 Generated with Claude Code