Repository navigation
feat(browser): split EXTRACT by content format so dispatch cannot swap HTML for text (#13236) - #13306
Merged
Merged
Conversation
…p HTML for text (#13236) The fourth contract gap the real callers exposed, and the one that would have caused a silent regression rather than an error. A single EXTRACT capability meant "return the page's text/markup", which conflated three different outputs -- and the three stacks each produce a different one: container render -> HTML worker get_text -> text, markup stripped in_process extract_content() -> text + structured_data web_fetch/_fetch_playwright needs HTML; it feeds a parser. Both the worker and the container declared EXTRACT and OUT_OF_PROCESS, and register_all registers the worker FIRST, so migrating web_fetch onto the interface would have resolved to the worker and returned text. Text is a valid result shape, so nothing would have raised -- the parser would just have received the wrong thing. Replaced with EXTRACT_TEXT / EXTRACT_HTML / EXTRACT_STRUCTURED, and each backend now declares what it actually returns rather than that it "extracts". ExtractRequest gains a ContentFormat, defaulting to TEXT. Requiring a capability at get_browser time and naming a format on the request are two statements that can disagree, so Browser.extract checks the resolved backend declares FORMAT_CAPABILITY[request.format] and raises UnsupportedFormatError instead of returning the wrong shape. ADR-009 is amended in the same commit rather than left to drift -- it still documented the single EXTRACT, and its step order nominated content_reach as the first migration, which the audit showed is wrong (it is the caller with the most stack-specific behaviour). web_fetch is the correct first migration. 44 tests, up from 36. The load-bearing one asserts that under the REAL registration order, web_fetch's requirement {EXTRACT_HTML, OUT_OF_PROCESS} resolves to the container and not the worker -- the exact swap this prevents.
…tainer (#13236) A fifth gap, found by checking whether the contract could actually serve web_fetch end to end after the format split. _fetch_playwright sends timeout: int(timeout * 1000) to the container's render endpoint. ExtractRequest had no timeout field -- NavigateRequest and ScreenshotRequest both do, so it was the odd one out -- and container.extract sent no timeout at all. Migrating web_fetch would therefore have silently dropped the caller's timeout and fallen back to the container default, with nothing failing to signal it. ExtractRequest gains timeout_seconds; the container converts to the milliseconds its endpoint expects, and omits the key entirely when the caller passes None so the container default still applies rather than timeout=0. 46 tests, up from 44.
Contributor
✅ SSOT Configuration Compliance: Passing🎉 No hardcoded values detected that have SSOT config equivalents! |
mrveiss
added a commit
that referenced
this pull request
Aug 2, 2026
…13236) (#13309) ADR-009 step 2 -- the first caller migrated off a directly-named stack, now that the five contract gaps the audit found are closed. _fetch_playwright imported services.playwright_service directly. It now declares what it actually needs and lets dispatch resolve it: requires={Capability.EXTRACT_HTML, Capability.OUT_OF_PROCESS} EXTRACT_HTML because the result feeds a parser, so text will not do. OUT_OF_PROCESS because the entire reason this fallback exists is that Playwright must not run inside the backend's own process. Behaviour-preserving, and that is not incidental. Both requirements are load-bearing: the worker and the container BOTH satisfy out-of-process extraction, and register_all registers the worker first, so under the pre-#13306 single EXTRACT capability this caller would have resolved to the worker and received get_text output -- plain text where a parser expects HTML, with nothing raising, because text is a valid result shape. The inline SSRF check is KEPT rather than deferred to the registry's guard. It is deliberate defence in depth: the container has network reach to private IPs, and one extra DNS resolution is cheaper than depending on a single chokepoint. The docstring says so, so a later reader does not remove it as redundant. Six tests pin that this lands on the same stack it called before, sends the same render payload with the same wait, forwards the caller's timeout, keeps returning None on failure so the render cascade can fall through, and that the worker is never selected despite registering first. 186 tests pass across autobot-backend/tests/unit/web_fetch, test_untrusted_web_content_boundary.py and autobot-backend/web_fetch. Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #13236
Thinking Path
The fourth gap the real callers exposed, and the only one that would have
produced a silent regression rather than an error.
A single
EXTRACTcapability meant "return the page's text/markup". Thatconflated three different outputs, and the three stacks each produce a
different one:
renderget_textextract_content()web_fetch/_fetch_playwrightneeds HTML — it feeds a parser. Both the workerand the container declared
EXTRACTandOUT_OF_PROCESS, andregister_allregisters the worker first. So migrating
web_fetchonto the interfacewould have resolved to the worker and returned text.
Nothing would have raised. Text is a valid
BrowserResult.content, so theparser would simply have received the wrong thing — the failure mode that
only shows up as bad output much later.
What Changed
EXTRACT→EXTRACT_TEXT/EXTRACT_HTML/EXTRACT_STRUCTURED. Eachbackend now declares what it actually returns rather than that it "extracts".
ExtractRequest.format(a newContentFormat, defaulting toTEXT).The two are tied together. Requiring a capability at
get_browsertimeand naming a format on the request are separate statements that can disagree,
so
Browser.extractchecks the resolved backend declaresFORMAT_CAPABILITY[request.format]and raisesUnsupportedFormatErrorratherthan returning the wrong shape.
ADR-009 amended in the same commit, not left to drift — it still
documented the single
EXTRACT, and its step order nominatedcontent_reachas the first migration. The audit showed that is wrong:
content_reachis thecaller with the most stack-specific behaviour (structured extraction plus
blocked_by_guardhandling).web_fetchis the correct first migration, andthe amendment says so.
A fifth gap, found by checking the contract end-to-end
After the split I checked whether
web_fetchcould actually be served, and itstill could not.
_fetch_playwrightsendstimeout: int(timeout * 1000)to the render endpoint, butExtractRequesthad no timeout field —
NavigateRequestandScreenshotRequestboth do,so it was the odd one out — and
container.extractsent no timeout at all.Migrating would have silently dropped the caller's timeout and fallen back to
the container default.
ExtractRequest.timeout_secondsnow exists, thecontainer converts to the milliseconds its endpoint expects, and omits the key
entirely when the caller passes
Noneso the container default still appliesrather than
timeout=0.Verification
The load-bearing test uses the real backends under the real
registration order:
The others pin that a mismatched format raises before the backend is
reached (asserted against its call log), that every
ContentFormatmaps to acapability so a new format cannot
KeyErrorat dispatch, and that eachstack's declared formats match what its code actually returns.
I also checked no bare
Capability.EXTRACTsurvives anywhere, including thepackage docstring example —
grepclean.What this does not do
tool_handlermigration.web_fetch) is now genuinely unblocked and behaviour-preserving:{EXTRACT_HTML, OUT_OF_PROCESS}resolves to the container — exactly what_fetch_playwrightcalls today — and its timeout survives the move.Model Used
Claude Opus 5 (1M context)