feat(kb): generate a domain-specific retrieval eval set from AutoBot's own corpora #13248

Problem

The retrieval ground truth is 20 hand-written synthetic documents and a hardcoded query→doc-ID
dictionary. It contains no AutoBot content, so it cannot detect a regression on the corpora we
actually serve (our own docs, indexed source code, chat history, ingested knowledge). Hand-writing
a real ground-truth set is expensive enough that it will not happen, which is why the fixture has
stayed synthetic.

Without a domain-specific eval set, every other measurement in this umbrella is scored on the wrong
corpus.

Evidence

• autobot-backend/knowledge/rag_benchmarks.py:1106-1235 — TestRealKBBenchmarks seeds from
  _TOPIC_DOCS: 20 hand-authored documents across python/database/networking/ml topics.
• autobot-backend/knowledge/rag_benchmarks.py:776,844 — _GROUND_TRUTH is a literal dict mapping
  query strings to expected doc-ID sets; every entry is hand-written.
• autobot-backend/knowledge/rag_benchmarks.py:1017-1027 —
  BenchmarkDataset.from_ground_truth() already accepts an arbitrary ground-truth mapping and
  derives a deterministic dev/test split from it, so the ingestion point for a generated set
  already exists. This issue supplies the generator, not new harness plumbing.

Proposed approach

A generator that turns any AutoBot corpus into a labelled eval set:

1. For each source document, prompt an LLM to produce N questions, each paired with the verbatim
   excerpt from that document that answers it. Verbatim matters — the excerpt is the ground-truth
   span the span-metrics issue scores against.
2. Verify each excerpt actually occurs in its source document; drop pairs where it does not
   (models paraphrase). This check is cheap and catches the most common failure.
3. Quality filter: embed the question and its excerpt, drop the pair when similarity falls below a
   configured threshold — this removes questions the excerpt does not really answer.
4. Deduplicate: embed all questions for a corpus and greedily drop any question too similar to one
   already kept, so the set does not become twenty rephrasings of one question.
5. Emit in BenchmarkDataset.from_ground_truth()'s input shape, carrying both doc IDs (for the
   existing precision@k) and excerpt spans (for the new span metrics).

Both thresholds must be configuration, not literals, and the generator must record which model and
thresholds produced a given set — an eval set whose provenance is unknown cannot be trusted later.

Risks / constraints

• Generated questions inherit the generator model's blind spots. Treat the set as a regression
  detector, not ground truth about absolute quality, and say so wherever scores are reported.
• Set size drives statistical power directly. The current fixture's per-query variance is high; a
  set of ten questions cannot support a keep/discard decision. Target a size that supports the
  significance margin defined in the sweep-runner issue, and record the achieved size.
• Generation costs one LLM call per document plus embeddings for filtering — bound the corpus and
  cache by document hash so regenerating is not a full re-spend.
• Generated sets must never contain secrets, credentials, or internal hostnames lifted verbatim
  from an indexed corpus. Screen output before any set is committed.

Acceptance criteria

• Generator produces question + verbatim-excerpt pairs from a configured AutoBot corpus.
• Excerpt-occurs-in-source verification, with a test proving a paraphrased excerpt is dropped.
• Question/excerpt similarity filter and near-duplicate question filter, both threshold-driven
  from config, both unit-tested.
• Output loads directly into BenchmarkDataset.from_ground_truth() and produces a non-empty
  dev/test split, shown by run output.
• Provenance recorded with each generated set (source corpus, model, thresholds, date, size).
• Secret/hostname screening applied before any generated set is committed.