Skip to content

fix(cli): execute network-capable adapters in Node to bypass CORS#45

Open
osen77 wants to merge 1 commit intoepiral:mainfrom
osen77:fix/network-adapter-cors
Open

fix(cli): execute network-capable adapters in Node to bypass CORS#45
osen77 wants to merge 1 commit intoepiral:mainfrom
osen77:fix/network-adapter-cors

Conversation

@osen77
Copy link
Copy Markdown

@osen77 osen77 commented Mar 16, 2026

Summary

  • Site adapters with capabilities: ["network"] (e.g. hackernews/top, v2ex/hot) make cross-origin API calls that fail with TypeError: Failed to fetch when executed via CDP Runtime.evaluate in page context, due to CORS/CSP restrictions
  • Fix: detect capabilities: ["network"] in adapter metadata and execute the script directly in the Node process via new Function(), where fetch() has no CORS restrictions
  • Adapters without the "network" capability continue to run via CDP in page context, preserving cookie-based auth

Test plan

  • bb-browser site hackernews/top 3 — returns JSON (was failing with CORS error)
  • bb-browser site v2ex/hot — returns JSON (cross-origin API)
  • bb-browser eval "document.title" — still works via CDP (regression check)
  • bb-browser site zhihu/hot — runs via CDP with login cookies (no network capability)

Fixes #41

🤖 Generated with Claude Code

Site adapters with `capabilities: ["network"]` make cross-origin API
calls (e.g. hackernews/top calls Firebase API from a ycombinator page).
These calls fail with "TypeError: Failed to fetch" when executed via
CDP Runtime.evaluate in the page context, because the page's CORS/CSP
policy blocks cross-origin requests.

Fix: detect `capabilities: ["network"]` in adapter metadata and execute
the adapter script directly in the Node process using `new Function()`.
Node's fetch() has no CORS restrictions, so cross-origin API calls
succeed without needing browser context.

Adapters without the "network" capability continue to execute via CDP
in page context, preserving cookie-based authentication for sites that
require login (e.g. zhihu, weibo).

Fixes epiral#41
@yan5xu
Copy link
Copy Markdown
Collaborator

yan5xu commented Mar 18, 2026

Thanks for the contribution! The idea of running pure-API adapters in Node to bypass CORS/CSP is interesting, but there are some fundamental issues with the current approach that need to be addressed.

1. capabilities: ["network"] semantic ambiguity (critical)

The current implementation assumes "network" means "doesn't need browser context, safe to run in Node." But in practice, different adapter authors interpret "network" differently:

  • hackernews/top uses it to mean "calls public APIs" → works in Node ✅
  • twitter/user also has capabilities: ["network"], but its code uses document.cookie and relative paths like /i/api/graphql/...will crash in Node

This PR would silently break all adapters that have "network" but depend on browser context.

2. new Function() security concerns

Running adapter code via new Function() in the Node process gives it full access to process, require, fs, child processes, etc. The browser sandbox at least constrains what adapter code can do. If we go down this path, node:vm with a restricted context (only exposing fetch, JSON, console) would be much safer.

3. Missing timeout protection

The CDP execution path has WebSocket-level timeout, but await asyncFn() in Node can hang forever if an adapter's fetch never resolves. Need at minimum a Promise.race with a timeout.

4. Relationship with PR #15

PR #15 tackles the same issue (#41) from the browser side (CSP bypass + credentials downgrade). The two approaches are complementary — #15 as the general fix, and Node execution as an optimization for truly browser-independent adapters.

Suggestion

If we want to support Node execution, I'd recommend:

  1. New explicit field instead of reusing "network" — e.g., "runtime": "node" to clearly signal "this adapter has no browser dependencies"
  2. node:vm sandbox instead of new Function() — only expose fetch, JSON, console
  3. Add timeout via Promise.race

Since capabilities is a free-form string[] with no validation or documentation, we should define the vocabulary clearly before making runtime decisions based on it.

Happy to discuss the design further!

SamCuipogobongo pushed a commit to SamCuipogobongo/bb-browser that referenced this pull request Apr 3, 2026
…ble-web-security

Site adapters that make cross-origin API calls (e.g. hackernews/top fetching
from firebaseio.com, bbc/news fetching from feeds.bbci.co.uk) fail with
"TypeError: Failed to fetch" because Runtime.evaluate runs in page context
where browser CORS restrictions apply.

Two complementary fixes:

1. Add --disable-web-security to managed browser launch args. This flag only
   affects the dedicated bb-browser Chrome instance (separate user-data-dir),
   not the user's daily browser. On about:blank tabs (null origin), this
   completely eliminates CORS enforcement.

2. Route adapters with capabilities: ["network"] to an about:blank tab instead
   of the target domain tab. Combined with (1), cross-origin fetch() calls
   succeed without CORS restrictions. Adapters requiring cookies (same-origin)
   continue to run on the domain tab as before.

3. Auto-retry fallback: if any adapter fails with "Failed to fetch" on a
   domain tab, automatically retry on an about:blank tab. This catches
   adapters that make cross-origin calls but don't declare capabilities:
   ["network"].

Tested: hackernews/top, bbc/news, arxiv/search, stackoverflow/search,
wikipedia/summary — all return valid JSON after this fix.

Fixes epiral#41
Closes epiral#110 (跨域 fetch 部分)
Related: epiral#104, epiral#45

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Site adapters fail with 'Failed to fetch' while eval works fine

2 participants