perf(search): eliminate vec0 full-scan amplification from max-k widening #1738

Parent incident: #1708

Priority: Early independent fix with a query benchmark

Confirmed code gap

Lore pins sqlite-vec 0.1.9 and uses exact flat KNN. vector-query.ts nevertheless describes the index as sublinear. Knowledge/distillation post-filtering and temporal chunk/session filtering can rerun the search at k=4096 when the initial result under-fills. The pinned implementation scans qualifying chunks and repeatedly selects minima within each chunk, so a larger k can add substantial work beyond distance calculation.

This is a demonstrated scaling hazard in code, not proof that it owns every 34–38 s production vector sample.

Sources: Lore query strategies, pinned sqlite-vec.

Implementation plan

1. Add a focused real-sqlite-vec benchmark varying corpus size, partition size, requested k, archived/category attrition, chunks per message and session filtering. Compare initial-k and widened queries, measuring worker service time separately from end-to-end latency.
2. Correct the sublinear claim and document each query's actual scan scope and widening behavior.
3. Reduce repeated scans by applying valid existing project/session/tenant restrictions before scoring where supported, and by replacing blanket max-k retry with an explicit bounded candidate strategy.
4. Preserve legitimate cross-project/global recall and visibility rules. Do not push a semantic scope restriction that changes the requested search.
5. Establish an exact-reference quality check for eligible top-k/message dedup. If the new strategy intentionally permits under-filled or approximate results, expose that outcome and quantify the tradeoff rather than silently dropping older memories.

Acceptance criteria

• Benchmark reproduces at least one expensive widening case and reports before/after service time, scan count and returned eligible hits.
• Common under-filled searches do not automatically trigger an unexplained second k=4096 scan.
• Scoping, archive/category filtering, chunk collapse and deterministic ordering have adversarial fixtures.
• No hard recent-N history cap is introduced as a hidden recall-quality regression.
• Query/partition migration compatibility is tested if a schema change is actually needed.

Dependencies and boundary

Can start independently; reuse #1733 when ready. Keep the first PR limited to query strategy. A backend/ANN migration is a separate, evidence-gated decision.

Hard prerequisites

None. This issue can be implemented independently.