fix: bypass CORS for site adapters via about:blank execution#159
Open
SamCuipogobongo wants to merge 1 commit intoepiral:mainfrom
Open
fix: bypass CORS for site adapters via about:blank execution#159SamCuipogobongo wants to merge 1 commit intoepiral:mainfrom
SamCuipogobongo wants to merge 1 commit intoepiral:mainfrom
Conversation
…ble-web-security Site adapters that make cross-origin API calls (e.g. hackernews/top fetching from firebaseio.com, bbc/news fetching from feeds.bbci.co.uk) fail with "TypeError: Failed to fetch" because Runtime.evaluate runs in page context where browser CORS restrictions apply. Two complementary fixes: 1. Add --disable-web-security to managed browser launch args. This flag only affects the dedicated bb-browser Chrome instance (separate user-data-dir), not the user's daily browser. On about:blank tabs (null origin), this completely eliminates CORS enforcement. 2. Route adapters with capabilities: ["network"] to an about:blank tab instead of the target domain tab. Combined with (1), cross-origin fetch() calls succeed without CORS restrictions. Adapters requiring cookies (same-origin) continue to run on the domain tab as before. 3. Auto-retry fallback: if any adapter fails with "Failed to fetch" on a domain tab, automatically retry on an about:blank tab. This catches adapters that make cross-origin calls but don't declare capabilities: ["network"]. Tested: hackernews/top, bbc/news, arxiv/search, stackoverflow/search, wikipedia/summary — all return valid JSON after this fix. Fixes epiral#41 Closes epiral#110 (跨域 fetch 部分) Related: epiral#104, epiral#45 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Site adapters that make cross-origin API calls fail with
TypeError: Failed to fetchbecauseRuntime.evaluateruns in page context where browser CORS restrictions apply. This affects ~50% of adapters includinghackernews/top,bbc/news,arxiv/search,stackoverflow/search, etc.Root cause: After #132 removed the Extension and unified the daemon, the maintainer noted "CDP 层面处理了跨域 fetch", but
Runtime.evaluatestill executes JS in page context, which is subject to CORS. The issue was never actually fixed.Changes
Two files, two complementary fixes:
1.
cdp-discovery.ts— Add--disable-web-securityto managed browser (1 line)The managed browser is a dedicated Chrome instance with its own
--user-data-dir, completely isolated from the user's daily browser. Adding--disable-web-securityeliminates CORS enforcement onabout:blanktabs (null origin), enabling cross-originfetch()calls.2.
site.ts— Smart tab routing for adapters (51 lines)capabilities: ["network"](and without"cookie") are routed to anabout:blanktab instead of the target domain tab. Combined with--disable-web-security, cross-origin fetch succeeds."Failed to fetch"on a domain tab, automatically retry onabout:blank. This catches adapters that make cross-origin calls but don't declarecapabilities: ["network"](e.g.bbc/news,arxiv/search).Why not the other approaches?
Page.setBypassCSP)new Function()in Node)"network"capability is ambiguous — some adapters declare it but depend ondocument.cookie. Also exposesprocess/require/fsto adapter codeabout:blank+--disable-web-security)Test plan
Tested on macOS with Chrome 146 + bb-browser 0.11.2:
bb-browser site hackernews/top 3— ✅ (was: Failed to fetch)bb-browser site bbc/news— ✅ (was: Failed to fetch)bb-browser site arxiv/search "LLM agent"— ✅ (was: Failed to fetch)bb-browser site stackoverflow/search "async await"— ✅ (was: Failed to fetch)bb-browser site wikipedia/summary "Python"— ✅ (was: Failed to fetch)bb-browser eval "document.title"— ✅ regression checkbb-browser site zhihu/hot— needs login (expected, cookie-dependent adapter)Fixes #41
Partially addresses #110 (cross-origin fetch portion)
Related: #104, #45
🤖 Generated with Claude Code