perf(picker): stop rebuilding catalogs per model row when /model opens

[SparkofSpike]

Summary

Opening /model froze the whole TUI. On a machine with 52 configured
routes and 3,444 catalog rows it took 3-4 seconds; it now takes ~150 ms,
measured both before and after with temporary in-process marks that are
not part of this PR.

Four independent causes, each fixed in its own commit:

1. One relay's catalog was rebuilt once per model row. Three separate
seams each repeated work that depends only on the route:

• cached_entry_for_route returned a deep clone of the whole cached
  entry, so every row cloned every offering. On a 176-model relay that
  was 30,976 CatalogOffering copies for one picker open.
• catalog_offering_for_route then rebuilt the provider's owned
  bundled + Models.dev row map per row, cloning ~60 rows and allocating a
  fresh BTreeMap each time.
• resolve_for_model re-ran the credential resolver (secret store
  included) and the auth-class classification per row, and
  route_identity_for_model inside it ran the class walk a second time.

with_cached_entry_for_route lends the entry to a callback;
source_rows_for_provider is memoized on the live generation;
RouteReadinessScope hoists the route-scoped readiness facts.

2. route_is_valid_for_model re-projected the whole bundled catalog per
row. It called RouteResolver::new(), and new() = from_offerings( default_offerings()), which re-projects every text-chat row of the
committed Models.dev asset plus the corrections pass and the
de-duplication maps. On 176 rows that is 176 full catalog projections.
RouteResolver::bundled() now shares one OnceLock-guarded resolver;
new() and with_configured_models are unchanged for callers that add
route-specific offerings.

3. The auto row built a full ModelInventory for one line of text.
auto_picker_hint read four router facts and paid for every configured
route's every model through runtime route resolution: 308 ms of a
1048 ms build. The router block never reads candidates, so
ModelInventory::auto_router_only skips only the candidate loop. No
logic is duplicated.

4. offerings_for_provider_identity scanned the whole merged catalog
per row. It walked every offering and computed
catalog_partition_key -- a trim().to_string() -- for each one, so a
3,444-row build over a few thousand offerings spent tens of millions of
string allocations on provider lookups. This was the largest single
stage (662 ms of 757 ms). SharedSnapshot now carries a
provider -> rows index built once with the snapshot.

Measurements

Stage marks from the temporary instrumentation, same machine, opening
/model once per run:

stage	before	after
auto row	308 ms	1.1 ms
per-row metadata	663 ms	79 ms
per-row readiness	16.8 ms	13.5 ms
per-row receipt	12.8 ms	10.1 ms
per-row push	12.3 ms	4.7 ms
total picker build	1048 ms	149 ms

Per-route detail moved with it: openrouter (390 rows) 108.6 ms ->
17.0 ms, edenai (334 rows) 95.9 ms -> 17.8 ms, openai (177 rows)
48.0 ms -> 7.7 ms.

These numbers come from one machine and are quoted as an indication of
which seams mattered, not as a guarantee. The next bottleneck on a
large-route setup is cloud_facts::overlay::snapshot(), which takes the
process-wide overlay write lock once per row for every non-custom
provider; this PR does not touch it.

Correctness

Nothing here changes observable behavior; each commit is a caching or
hoisting change behind an existing seam.

• catalog_offering_for_route keeps all five outcomes exactly: listed
  (row cloned and completed from lower layers), fetched-but-unlisted
  (only an attested-unlisted row, never a fall-through to the bundled
  merge), fetched_at == 0, absent entry, and cache error.
• source_rows_for_provider's memo key is (live generation, cloud generation or a sentinel, live layer present). The generation is
  sampled before the caller reads LIVE_SNAPSHOT, the ordering
  merged_snapshot already documents, so a concurrent refresh cannot
  leave stale rows tagged with the current generation.
• SharedSnapshot's index uses the same catalog_partition_key
  derivation and preserves offering order, so offerings_for_provider
  returns the same rows in the same relative order as the scan it
  replaces.
• auto_router_only cannot change a router field: every one of them is
  derived from config after the loop, never from candidates.
• RouteResolver::bundled() is Self::new under a OnceLock, and
  new() has no environment, config, clock or network input, so it
  cannot freeze a stale answer. resolve takes &self.

Type of Change

• Performance (no behavior change)

Testing

• cargo test -p codewhale-tui --lib -- provider_lake model_picker provider_picker provider_readiness client::catalog_tests model_inventory pricing -> 411 passed; 0 failed; 0 ignored
• cargo test -p codewhale-config --lib -- route:: -> 102 passed; 0 failed; 1 ignored (commit 2 adds a public API in this crate)
• cargo check -p codewhale-tui --lib -> clean
• rustfmt --edition 2024 --check on all four changed files -> clean
• cargo build --release -p codewhale-cli -p codewhale-tui -> Finished
• The reported symptom was re-measured end to end on the reporting
  machine: 3-4 s before the chain, ~150 ms after.

Notes for review

• Two additions are worth calling out because they are not pure caching:
  a second permanently resident RouteResolver::bundled() projection
  (a few MB; the client already keeps one), and a 32-entry memo of
  provider partitions that is evicted alphabetically rather than by use.
• The SharedSnapshot index normalizes provider keys through
  catalog_partition_key (trimmed). No producer of padded provider
  strings was found, so this is a latent change, not an observed one.
• No new tests: project policy. The changed paths are covered by the
  existing suites above; a direct assertion that bundled() and new()
  resolve identically would be the one test I would add if asked.

Related Issues

No-Issue: reported as local /model latency; no upstream issue was opened.

Attribution

🤖 Generated by SpikeBot 000(CodeWhale-LOCAL)

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Hmbown]

🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

Thank you @SparkofSpike. You traced the /model freeze to four separate per-row rebuilds (the cloned cache entry, the Models.dev partition, the bundled resolver projection and the full-inventory Auto hint) and fixed each behind its existing seam, which made this easy to review. Reading the live generation before the snapshot, and keying routes without signed facts separately, were exactly right. Merged in e133977 for 0.10.2, which hasn't shipped yet. CI passed the full Linux and macOS suites, and Windows after re-running one unrelated plugin-state flake. The 3–4 s → ~150 ms timing is your measurement; we haven't reproduced it on a 52-route setup ourselves.