Support extracting login-gated sites + fix broken page.evaluate() under tsx - #1
Open
ulises2k wants to merge 1 commit into
Open
Support extracting login-gated sites + fix broken page.evaluate() under tsx#1ulises2k wants to merge 1 commit into
ulises2k wants to merge 1 commit into
Conversation
…aluate() under tsx - Add bin/login.ts: logs into a WordPress-style (or generic log/pwd) login form with Playwright and saves the resulting session as a storageState JSON file (cookies + localStorage). - Add --storage-state <path> to `pnpm engine:extract`, threaded through crawl.ts and extract.ts, so every browser context (crawl phase, DOM collection, dark-mode detection, framework detection) can be created pre-authenticated. Without this, extraction of any page behind a login wall was impossible. - Fix a pipeline-breaking bug unrelated to auth: tsx (the runtime behind `pnpm engine:*`) hardcodes esbuild's keepNames: true, which wraps every named function/const inside a module with __name(target, "name") calls. Playwright's page.evaluate(fn) ships fn.toString() into an isolated browser context that never sees the Node-side __name helper, so ANY evaluate callback with an inner named function/const (dom-collector.ts, css-analyzer.ts, interaction-capture.ts, etc. -- most of the engine) threw "ReferenceError: __name is not defined" and every extraction silently produced 0 colors / 0 typography / 0 components, on any site. lib/engine/patch-evaluate.ts patches Page.prototype.evaluate once (reached via any live page instance, since Playwright doesn't export the Page class) to rebuild the callback with a self-contained __name shim before Playwright serializes it, so all ~35 existing evaluate() call sites across the engine keep working unchanged. Verified end-to-end against a real login-gated WordPress/LearnDash site: login -> authenticated 12-page crawl (including /tu-cuenta/ member area) -> 6,808 elements extracted -> full DESIGN.md v2 generation, 98/100 validation score, 100% proof coverage.
|
@ulises2k is attempting to deploy a commit to the sunil negi's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for extracting design tokens from login-gated sites.
pnpm engine:extracthad no way to authenticate, so any site behind a login wall (member areas, dashboards, LMS platforms, etc.) was impossible to extract.bin/login.ts: logs in with Playwright (works with WordPress-stylelog/pwdforms — wp-login.php and Elementor Pro login widgets) and saves the session as a PlaywrightstorageStateJSON file.--storage-state <path>flag onpnpm engine:extract, threaded throughcrawl.tsandextract.tsso every browser context created during the pipeline (crawl phase, DOM collection, dark-mode detection, framework detection) is pre-authenticated.Fixes a pipeline-breaking bug, unrelated to auth, that affects every extraction on every site.
tsx(the runtime behindpnpm engine:*) hardcodes esbuild'skeepNames: true, which wraps every named function/const inside a module with__name(target, "name")calls. Playwright'spage.evaluate(fn)shipsfn.toString()into an isolated browser context that never sees the Node-side__namehelper, so any evaluate callback containing an inner named function or const — which is most of the engine (dom-collector.ts,css-analyzer.ts,interaction-capture.ts, etc.) — throwsReferenceError: __name is not defined. The practical effect: extraction currently silently returns 0 colors / 0 typography levels / 0 components for any URL on a fresh install with a recenttsx.lib/engine/patch-evaluate.tspatchesPage.prototype.evaluateonce (reached via any livepageinstance, since Playwright doesn't export thePageclass itself) to rebuild the callback with a small, self-contained__nameshim inlined into its own body before Playwright serializes it. All existingpage.evaluate()call sites across the engine (~35 of them, in 8 files) keep working completely unchanged — no call-site edits needed.Verification
Ran the full pipeline end-to-end against a real login-gated WordPress + LearnDash site: login → authenticated 12-page crawl (including the member-only
/tu-cuenta/area) → 6,808 elements extracted, 62 colors, 146 typography levels, 8 component types → full DESIGN.md v2 generation → 98/100 validation score → 100% proof coverage.Also reproduced the
__namebug in isolation onhttps://example.comwith a stock--fastextraction (no auth involved) to confirm it's a general regression, not something specific to the auth changes — before the fix: 0 colors/0 typography/0 components; after: 3 colors/2 typography/1 component as expected for that page.Notes
bin/login.tstakes the login URL/email/password as CLI args (nothing hardcoded), and/output/is already gitignored.--storage-stateis optional everywhere it was added; omitting it preserves current unauthenticated behavior exactly.