feat(config): let the LightPanda escalation target a renderer that exists

[rqi14]

The dead end

scrape_url_inner escalates after a thin LightPanda body and hardcodes the next tier:

https://github.com/us/crw/blob/c74bcf3/crates/crw-crawl/src/single.rs#L456-L462

let escalation_target: Option<&str> = if prior_renderer == Some("lightpanda") {
    Some("chrome")
} else {
    pinned
};

The pool has no tolerance for a name it does not hold — a pinned renderer that is absent is a hard error, not a fallback:

https://github.com/us/crw/blob/c74bcf3/crates/crw-renderer/src/lib.rs#L2401

"requested renderer '{}' not in pool [{}]"

Those two compose into a dead end on any deployment that runs no Chrome CDP sidecar. Every post-LightPanda escalation is pinned to a tier that cannot be constructed, so it fails on the pin rather than on the page — and stronger tiers that are configured and healthy (camoufox in particular) are never reached. The else arm already defers to the chain so the http tier can find chrome through the normal failover path; only the LightPanda arm is nailed shut.

The change

extraction.lightpanda_escalation_renderer, #[serde(default)] → "chrome". An install that does not set it is byte-identical to today; the only behaviour that changes is for a config that opts in. Env: CRW_EXTRACTION__LIGHTPANDA_ESCALATION_RENDERER.

Why this is not visible upstream

Same shape as #485: a dead path that only a particular deployment form exposes. If you run the Chrome sidecar, the hardcoded name is always correct and the branch is invisible. It is only wrong for installs whose ladder tops out somewhere else, and those installs cannot report it as a renderer bug because the error surfaces as a pool-membership error, not a scrape failure.

Tests

The sister key lightpanda_retry_threshold_bytes is covered by a default assertion and a TOML-override assertion; the new key gets one of each, in the same two tests:

• config::tests::extraction_config_defaults — asserts the default is "chrome"
• config::tests::extraction_config_toml_scalar_overrides — asserts a TOML override lands

$ cargo test -p crw-core
test result: ok. 499 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 13 passed; 0 failed (config_tests)
test result: ok. 2 passed; 0 failed (api_casing)
test result: ok. 3 passed; 0 failed (error_tests)
test result: ok. 21 passed; 0 failed (types_tests)

$ cargo fmt --check   # clean
$ cargo clippy --workspace --all-targets   # no new warnings

cargo test -p crw-crawl --lib is green except pdf::tests::convert_pdf_bytes_* (2), which fail identically on unmodified c74bcf3 on this machine — a local pdfium/native-lib issue on Windows, not related to this change.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[us]

Thanks for this one, and for the write-up. The diagnosis is right, and it is worse than the PR
says: the repo's own shipped default is one of the broken configs. config.default.toml sets
[renderer.lightpanda] with [renderer.chrome] commented out, that file is a real load layer
and is COPY'd into the image, and docs/docs/configuration.md documents
docker run -e CRW_RENDERER__MODE=lightpanda ghcr.io/us/crw:latest with no CRW_CONFIG. That
container has pool ["lightpanda"], so every post-LightPanda escalation dies on the pin and
the caller gets JS escalation failed: requested renderer 'chrome' not in pool [lightpanda]
concatenated into the response warning.

I have pushed a commit onto your branch that takes it a different way, and I want to explain
why rather than just landing it.

There is exactly one correct answer at that line, "the strongest tier this deployment actually
has", and the runtime can compute it. A key that defaults to the same literal fixes no
deployment until an operator finds it, and it is documented in neither config.default.toml
nor docs/, so the install that hits this has no way to learn the key exists. Your own
function already calls renderer.js_renderer_names() about 300 lines earlier to reject an
unavailable user pin, so the pool was already in reach.

What is on the branch now:

// crw-renderer: the tiers the auto chain may enter on its own
pub fn auto_ladder_names(&self) -> Vec<&str>
// crw-renderer: which tier a post-LightPanda escalation should aim at
pub fn lightpanda_escalation_target(&self) -> Option<&str>

// crw-crawl/src/single.rs
let escalation_target: Option<&str> = if prior_renderer == Some("lightpanda") {
    renderer.lightpanda_escalation_target()
} else {
    pinned
};

Three details that took a second review round to get right, and are worth calling out because
the naive version of this is wrong in two ways:

1. chrome stays the first choice, not "first non-lightpanda in the list". The ladder is
   reordered at request time for a host the preference learner has promoted to chrome, so
   position alone is not enough: a [lightpanda, playwright, chrome] pool on a promoted host
   reaches chrome today, and picking by position would have sent it to playwright with no
   failover, because a pin filters the pool to one entry. Preferring chrome keeps every pool
   that holds it behaving exactly as before.
2. chrome_proxy is excluded while auto_egress_escalation is on. The chain deliberately
   lifts it out of the ladder and fires it only on a hard block, load-shed, with its own
   budget. Naming it as an escalation target sets is_user_pinned and skips all of that, which
   would put a paid residential render on every thin page. The in-repo note says that
   configuration measured at success −2pp and p90 +69%.
3. camoufox held out by include_in_auto = false is excluded, because pinning it by name
   bypasses the exclusion, and an internally chosen escalation must not override a deliberate
   opt-out.

None now means the escalation is skipped rather than dispatched. Falling through to "auto"
would re-render lightpanda for the same thin result, which is what your own retained comment
warns about.

Tests, all passing:

test tests::lightpanda_escalation_target_picks_a_tier_the_pool_actually_holds ... ok
  lightpanda + chrome + chrome_proxy      -> "chrome"        (production shape, unchanged)
  lightpanda + chrome_proxy               -> "chrome_proxy"  (was a dead end)
  lightpanda + playwright                 -> "playwright"    (was a dead end)
  lightpanda only                         -> None            (no unsatisfiable pin)
  lightpanda + playwright + chrome        -> "chrome"        (position does not decide)
  lightpanda + chrome_proxy, auto_egress  -> None            (gate not bypassed)

test tests::auto_ladder_names_respects_the_camoufox_opt_out ... ok

cargo test --workspace is green, and cargo clippy --workspace --all-targets -- -D warnings
is clean. The one failure I see locally is http_only::tests::is_pdf_check_is_case_insensitive,
a local-port bind that also fails on unmodified main on this machine and passes in isolation.

The config key is gone from the branch. If you would still like an operator override on top,
it needs to be typed rather than a bare String ("lightpanda" re-runs LightPanda, "auto"
silently does the same, "" and any typo surface as requested renderer '' not in pool [...]
on a customer response, and "cloak" can never work because that arm is held outside
js_renderers), moved under [renderer] where every other renderer identity lives, and
documented. Happy to take that as a follow-up if you want it.

One housekeeping note: the branch now carries your feat(config) commit and my fix(renderer)
commit, and the first describes a key the branch no longer has. Worth collapsing before this
lands so the changelog does not announce it.