Skip to content

Support extracting login-gated sites + fix broken page.evaluate() under tsx - #1

Open
ulises2k wants to merge 1 commit into
sunil-dsb:mainfrom
ulises2k:feat/authenticated-site-extraction
Open

Support extracting login-gated sites + fix broken page.evaluate() under tsx#1
ulises2k wants to merge 1 commit into
sunil-dsb:mainfrom
ulises2k:feat/authenticated-site-extraction

Conversation

@ulises2k

@ulises2k ulises2k commented Sep 1, 2026

Copy link
Copy Markdown

Summary

  • Adds support for extracting design tokens from login-gated sites. pnpm engine:extract had no way to authenticate, so any site behind a login wall (member areas, dashboards, LMS platforms, etc.) was impossible to extract.

    • New bin/login.ts: logs in with Playwright (works with WordPress-style log/pwd forms — wp-login.php and Elementor Pro login widgets) and saves the session as a Playwright storageState JSON file.
    • New --storage-state <path> flag on pnpm engine:extract, threaded through crawl.ts and extract.ts so every browser context created during the pipeline (crawl phase, DOM collection, dark-mode detection, framework detection) is pre-authenticated.
  • Fixes a pipeline-breaking bug, unrelated to auth, that affects every extraction on every site. tsx (the runtime behind pnpm engine:*) hardcodes esbuild's keepNames: true, which wraps every named function/const inside a module with __name(target, "name") calls. Playwright's page.evaluate(fn) ships fn.toString() into an isolated browser context that never sees the Node-side __name helper, so any evaluate callback containing an inner named function or const — which is most of the engine (dom-collector.ts, css-analyzer.ts, interaction-capture.ts, etc.) — throws ReferenceError: __name is not defined. The practical effect: extraction currently silently returns 0 colors / 0 typography levels / 0 components for any URL on a fresh install with a recent tsx.

    • New lib/engine/patch-evaluate.ts patches Page.prototype.evaluate once (reached via any live page instance, since Playwright doesn't export the Page class itself) to rebuild the callback with a small, self-contained __name shim inlined into its own body before Playwright serializes it. All existing page.evaluate() call sites across the engine (~35 of them, in 8 files) keep working completely unchanged — no call-site edits needed.

Verification

Ran the full pipeline end-to-end against a real login-gated WordPress + LearnDash site: login → authenticated 12-page crawl (including the member-only /tu-cuenta/ area) → 6,808 elements extracted, 62 colors, 146 typography levels, 8 component types → full DESIGN.md v2 generation → 98/100 validation score → 100% proof coverage.

Also reproduced the __name bug in isolation on https://example.com with a stock --fast extraction (no auth involved) to confirm it's a general regression, not something specific to the auth changes — before the fix: 0 colors/0 typography/0 components; after: 3 colors/2 typography/1 component as expected for that page.

Notes

  • No secrets, credentials, or extraction output are included — bin/login.ts takes the login URL/email/password as CLI args (nothing hardcoded), and /output/ is already gitignored.
  • Backward compatible: --storage-state is optional everywhere it was added; omitting it preserves current unauthenticated behavior exactly.

…aluate() under tsx

- Add bin/login.ts: logs into a WordPress-style (or generic log/pwd) login
  form with Playwright and saves the resulting session as a storageState
  JSON file (cookies + localStorage).
- Add --storage-state <path> to `pnpm engine:extract`, threaded through
  crawl.ts and extract.ts, so every browser context (crawl phase, DOM
  collection, dark-mode detection, framework detection) can be created
  pre-authenticated. Without this, extraction of any page behind a login
  wall was impossible.
- Fix a pipeline-breaking bug unrelated to auth: tsx (the runtime behind
  `pnpm engine:*`) hardcodes esbuild's keepNames: true, which wraps every
  named function/const inside a module with __name(target, "name") calls.
  Playwright's page.evaluate(fn) ships fn.toString() into an isolated
  browser context that never sees the Node-side __name helper, so ANY
  evaluate callback with an inner named function/const (dom-collector.ts,
  css-analyzer.ts, interaction-capture.ts, etc. -- most of the engine)
  threw "ReferenceError: __name is not defined" and every extraction
  silently produced 0 colors / 0 typography / 0 components, on any site.
  lib/engine/patch-evaluate.ts patches Page.prototype.evaluate once
  (reached via any live page instance, since Playwright doesn't export the
  Page class) to rebuild the callback with a self-contained __name shim
  before Playwright serializes it, so all ~35 existing evaluate() call
  sites across the engine keep working unchanged.

Verified end-to-end against a real login-gated WordPress/LearnDash site:
login -> authenticated 12-page crawl (including /tu-cuenta/ member area)
-> 6,808 elements extracted -> full DESIGN.md v2 generation, 98/100
validation score, 100% proof coverage.
@vercel

vercel Bot commented Sep 1, 2026

Copy link
Copy Markdown

@ulises2k is attempting to deploy a commit to the sunil negi's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant