Repository navigation
Reranker test plan: Postgres parity, live-provider smoke, quality regression harness (post-#1143) #1231
Description
Context
PR #1143 (ironhalik) added the optional cross-encoder rerank stage: RerankProvider
protocol, FastEmbed (local ONNX) and LiteLLM (hosted) providers, factory with process-wide
singleton cache, opt-in config under reranker_*, fixed-prefix + demoted-tail pagination
stability, and fail-fast 503/502 error mapping. Unit coverage from the PR is strong
(pipeline math, provider contracts, factory, router mapping, redaction, config validation —
all in the default CI suites).
An audit of the merged feature found the remaining gaps concentrated in four areas, plus a
few small code fixes. This issue tracks closing them.
Phase 1 — CI-visible coverage gaps (one PR, no external deps)
- Postgres parity: the 9 repository-integration rerank tests in
tests/repository/test_rerank_pipeline.py(:527-841) skip on Postgres
("sqlite-vec repository behavior is local SQLite-only"), so hybrid/vector rerank
ordering on pgvector is asserted only by the single semantic-marked benchmark test.
Port them to a Postgres variant (recording-reranker fixture;test-int/semantic/+
semanticmarker if they need real vector tables, main suite otherwise). - FTS negative guard: provider configured + FTS/title/permalink search → provider
never invoked, ranking identical to baseline. (Only the inverse — no provider — is
tested today:test_search_without_reranker_keeps_baseline.) - Wiring seams:
create_search_repositoryinjects the singleton provider into both
backends (search_repository.py:175-207); repository__init__self-resolution
(sqlite_search_repository.py:84-86,postgres_search_repository.py:100-102) respects
semantic_search_enabled. - Config-to-pipeline:
reranker_max_document_charsset via config truncates the
document text the provider receives (currently only the pure helper is tested). - ChatGPT adapter opacity:
chatgpt_tools.py:205's broadexcept Exception
collapses rerank 503s into a generic "Internal search error". Distinguish retryable
unavailability in the returned payload, and pin it with a test.
Small fixes (ride the Phase 1 PR as separate commits)
- Stale constructor default in
fastembed_rerank_provider.py:71
(Xenova/ms-marco-MiniLM-L-6-v2; the real default isjinaai/jina-reranker-v1-tiny-en
fromconfig_models.py:61) — dead today but a trap for direct construction. -
reranker_timeoutconfig key: the LiteLLM provider hardcodestimeout=30.0
(litellm_rerank_provider.py:34) and the factory never passes it. Add the config field
(validatedgt=0, default 30), thread it through the factory, document it in
docs/semantic-search.md.
Phase 2 — real-model and live-provider verification
- Real-FastEmbed smoke (semantic-marked, runs in the
test-semanticCI job with model
caching): load actualjinaai/jina-reranker-v1-tiny-en, tiny fixed corpus, assert the
relevant doc ranks first and scores are[0,1]-calibrated. - Extend
test-int/semantic/litellm_live_harness.py(embeddings-only today) with
rerank: Coherererank-v3.5+ one alternate (Jina or Voyage), env-keyed, run manually
with real keys — contract shape, ordering sanity, auth-failure classification
(permanent vs transient). Mirrors the feat(core): add LiteLLM embedding provider #809 embedding live-smoke pattern.
Phase 3 — quality regression harness
- Golden-query eval on a realistic corpus (the Moby Dick demo project, 221 notes):
~20 queries with expected top notes; measure hit@k and MRR reranker-off vs reranker-on
(FastEmbed). Build on the existingtest-int/semantic/metrics.py/corpus.py/
report.pymachinery. Assert no regression; report the delta. This is the durable guard
the LoCoMo motivation (Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950) asked for, and the baseline for Add reranking step to search pipeline (local cross-encoder) #618/Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666/Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark #951 follow-ons. - Latency benchmark (benchmark-marked): vector + hybrid P50/P95 with/without the local
reranker — validates the "~20x headroom" claim and catches future model swaps that blow
the budget.
Phase 4 — operational drills (manual, test-live style)
- Config UX: enable without
semantic_search_enabled→ clear error at
bm config settime; model typo rejected at set-time; secrets absent from
config list/ diagnostics output. - Live failure drill: kill network mid-query → API 503; multi-project MCP aggregate
aborts rather than returning a partial page; recovery without restart. - Pagination stability under a live provider: page 1 → concurrent writes → page 2,
no duplicates or omissions.
Notes for implementers
- Reranking applies to vector and hybrid modes only; FTS/title/permalink keep their
ranking (search_repository_base.py:1646). Scores replace fused scores for the fixed
20-candidate prefix; the tail is demoted rank-only (rerank_provider.py:31-42) — the
zero-floor tie-break is order-dependent, so downstream score-only re-sorts would break
pagination stability. - Failure semantics are deliberately fail-fast (no silent un-reranked fallback): a
degraded page could duplicate/omit results relative to an already-served reranked page.
Tests must not "fix" this.
Refs: #1143 (feature), #950 (motivation, closed), #618 #666 #951 #1155 (open follow-ons).
Activity
claude commentedon Aug 11, 2026
Claude finished @phernandez's task in 53s —— View job
Triage
Type: Enhancement — test/QA coverage work for the reranker feature shipped in #1143, not a defect report.
Priority: Medium. No user-facing breakage; closes verification gaps (Postgres ordering parity, wiring seams, live-provider/quality regression coverage) for an opt-in feature that already has strong unit coverage.
Component: Search/semantic-search reranker (
RerankProvider,search_repository.py,chatgpt_tools.py) and its test suites (tests/repository/,test-int/semantic/). Touches Postgres integration test paths used by cloud deployments, but isn't itself Cloud-specific behavior, so no cloud label applied.Complexity: Mixed and clearly staged by the issue itself — Phase 1 (Postgres parity tests, FTS negative guard, wiring/config tests, two small code fixes) is medium and self-contained; Phase 2 (live-provider smoke) and Phase 3 (golden-query quality harness + latency benchmark) are larger and depend on external model/API access; Phase 4 is manual operational drills, not automatable CI work.
Duplicate check: No exact duplicate found among the issue's own cross-references (#618, #666, #950, #951, #1143, #1155) — this is a new tracking issue consolidating post-#1143 audit gaps rather than restating any of them.
Labels applied:
enhancement.phernandez commentedon Aug 11, 2026
Phase 1 + small fixes are complete — delivered by #1232 (merged). All boxes ticked.
Scope decision for the remaining phases:
instance, run searches, page through). The drills as originally listed re-verify behavior
that now has automated coverage on both backends: config validation + secret redaction,
503/502 mapping, multi-project aggregate abort, and cross-page pagination stability.
recommendation or a cloud tier). It remains the one manual item with unique value — real
provider response shapes through
litellm.arerankcan't be validated by stubs (samerationale as the feat(core): add LiteLLM embedding provider #809 embedding live smoke).
work, not manual testing — Phase 3 is the gate for revisiting Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark #951 and closing Add reranking step to search pipeline (local cross-encoder) #618.
phernandez commentedon Aug 11, 2026
Measured baseline (Phases 2-3, PR #1233)
Recorded here so the tracker carries the numbers, not just the PR description. All figures
from the real models (
bge-small-en-v1.5embeddings +jinaai/jina-reranker-v1-tiny-encross-encoder) on the curated golden ranking set; the harness re-prints these on every run
(
test-int/semantic/test_reranker_quality.py,test_reranker_latency.py).Ranking quality (golden set, curated ranking-sensitive queries)
No queries excluded; the tiny model degraded nothing. The suite asserts on ≥ off for
hit@1/MRR plus a hard hit@5 floor, so a silent rerank no-op fails CI.
Latency (local, warmed, 20 iterations)
≈90 ms P50 overhead for the local tiny cross-encoder — inside the latency headroom that
motivated #950. The benchmark test enforces only a generous catastrophic-regression
ceiling; these figures are the reference point.
These numbers are the baseline for the #951 revisit (segment LoCoMo single-hop misses into
rerank-recovered vs absent-from-pool) and for closing #618 once the benchmark evidence is
accepted.
phernandez commentedon Aug 11, 2026
LoCoMo rerank comparison complete — full results on #951 (#951 (comment)). Headline: overall R@5 +5.0pp / MRR +9.7pp with reranking on; single_hop (the entity-confusion category) gained most at +9.3pp R@5 / +11.9pp MRR. Latency caveat recorded there: p95 420ms → ~2.1s on the long-chunk LoCoMo corpus with the full ~100-candidate pool — mitigations are reranker_max_document_chars and a smaller pool. This closes the quality-evidence loop for #618 as well.
phernandez commentedon Aug 11, 2026
Closing: everything actionable is delivered — Phase 1 + fixes (#1232), real-model smoke + quality harness + latency benchmark (#1233), and the LoCoMo before/after comparison recorded here and on #951 (overall R@5 +5.0pp / MRR +9.7pp; single_hop +9.3pp R@5). The hosted Cohere/Jina live check is dropped — not deferred — as there's no interest in hosted rerank providers; if that ever changes, the harness extension sketch lives in this issue's history. Only informal loose end: an optional ten-minute smoke with reranker_enabled on a personal instance. Follow-up filed for the one data-backed finding that deserves its own issue: reranker latency on long-document corpora.