Repository navigation
feat: support evaluating scripts from local files - #2606
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
2616476 to
945ef28
Compare
cbi58105-crypto
left a comment
There was a problem hiding this comment.
Summary: Adds sourcePath + format to evaluate_script allowing loading local files; supports 'function' and 'script' modes; validates exactly one source and disallows args when format='script'. Tests added covering file loading and error cases. Suggestions: clarify how relative sourcePath is resolved (process.cwd vs repo root) and document file: URL handling; consider clearer error messages for unreadable paths; note ESM modules are not supported. I couldn't run tests locally due to environment (PowerShell execution policy / missing tsc).
OrKoN
left a comment
There was a problem hiding this comment.
Issues & Potential Bugs
1. format: "function" Wrapping Bug with Single-Line Comments and Trailing Semicolons
- Location:
src/tools/script.ts:328(using fn = await evaluatable.evaluateHandle(\(${source})`);`) - Problem:
- Single-line comments at EOF: If a file loaded via
sourcePath(or inlinefunction) ends with a single-line comment without a trailing newline (e.g.() => document.title // get title), wrapping without newlines produces(() => document.title // get title). The closing parenthesis)is commented out, throwingSyntaxError: Unexpected end of input. Wrapping with newlines ((\n${source}\n)) prevents this. - Trailing semicolons in formatted
.jsfiles: Becauseformatdefaults to'function'even whensourcePathis provided, loading a standard.jsfile formatted by Prettier/ESLint (which appends a trailing semicolon, e.g.() => { return 1; };) produces(() => { return 1; };), throwingSyntaxError: Unexpected token ';'.
- Single-line comments at EOF: If a file loaded via
- Recommendation: Wrap with newlines (
(\n${source}\n)), trim trailing semicolons/whitespace whenformat === 'function', or consider defaultingformatto'script'whensourcePathis used without an explicitformat.
2. Serialization Discrepancy Between format: "function" and format: "script"
- Location:
src/tools/script.ts:327-340(performEvaluation) - Problem:
- In
format: "function",JSON.stringify(await fn(...args))executes inside the browser context. - In
format: "script",const value = await evaluatable.evaluate(source)relies on CDPRuntime.evaluate({ returnByValue: true })to transfer the value to Node.js, and then callsJSON.stringify(value)in Node.js. - CDP's C++
returnByValueserializer does not invoke.toJSON()and ignores prototype getters. Consequently, browser objects likeDOMRect(el.getBoundingClientRect()),PerformanceEntry, orURLserialize completely informat: "function"({"x":0,"y":0,"width":100,...}) but serialize to{}informat: "script". - Additionally, if a script evaluates to a non-serializable object (e.g. a DOM node or
window), CDP throws a protocol error (Could not serialize object/Object reference chain is too long).
- In
3. Relative sourcePath Resolution Bug in CLI Daemon Architecture
- Location:
src/bin/chrome-devtools.ts:288-302&skills/chrome-devtools-cli/SKILL.md:63 - Problem:
SKILL.mdrecommends running:
chrome-devtools evaluate_script --pageId 1 --sourcePath ./script.js --format script- In
src/bin/chrome-devtools.ts, the CLI client forwardscommandArgs(including relative paths like./script.js) verbatim over IPC to the backgroundchrome-devtools-mcpdaemon process. - If the background daemon was started in a different working directory than the user's current shell
cwd,resolveScriptSourceresolves./script.jsrelative to the daemon's working directory rather than the CLI caller's working directory, resulting inUnable to read script sourceor reading the wrong file. - Note: The E2E test in
tests/e2e/chrome-devtools-commands.test.ts:396passes an absolute path (path.join(os.tmpdir(), 'script.js')), which masked this bug.
- Recommendation: Resolve relative file paths (
sourcePath,filePath) againstprocess.cwd()in the CLI client (src/bin/chrome-devtools.ts) before forwarding the command payload to the daemon, or pass the client'scwdto the daemon.
Testing & Coverage Issues
4. Violation of AGENTS.md Testing Rules (withMcpContext Used for Unit Validation Tests)
- Location:
tests/tools/script.test.ts:511-562 - Problem:
AGENTS.mdspecifies:Prefer mock-based unit tests over real-browser tests: Do not use
withMcpContextor launch a real browser unless the test genuinely requires real browser or DevTools protocol integration...- The new tests
requires exactly one script source,rejects args for classic scripts, andreports unreadable source filesall launch a real Chrome browser viawithMcpContextsolely to assert synchronous/early parameter validation errors that throw before touchingcontext,page, or Puppeteer.
- Recommendation: Use
createHandlerMocks()fromtests/mocks.tsfor handler validation unit tests.
5. Missing Test Coverage for verifyFilesSchema (sourcePath: true) and Relative CLI Paths
- Problem:
- All tests in
tests/tools/script.test.tscallevaluateScript().handler(...)directly, bypassingToolHandler.handle()whereverifyFilesSchema(validateToolFilesinsrc/ToolHandler.ts) validates paths againstcontext.validatePath(). There is no test verifying thatsourcePathis checked byToolHandleror thatfile://URLs rewritten byToolHandlerare handled end-to-end. tests/e2e/chrome-devtools-commands.test.tsonly tests an absolutesourcePath; adding an E2E test with a relative--sourcePath ./script.jsfrom a different working directory will catch the CLI/daemon CWD mismatch described in Issue #3.
- All tests in
945ef28 to
98e8169
Compare
|
@OrKoN I've addressed the review feedback and rebased the PR onto the latest main.
Validation: npm run gen; targeted script, ToolHandler, CLI E2E, daemon, and pages tests pass. The full suite only retains the existing Windows-specific large full-page screenshot failure (Page.captureScreenshot: Page is too large). Please take another look when convenient. |
98e8169 to
bda681f
Compare
bda681f to
27830bd
Compare
OrKoN
left a comment
There was a problem hiding this comment.
I think we need a few more changes:
- Keep
sourcewrapped in parentheses ((\n${source}\n), stripping an optional trailing semicolon first) instead of0,\n${source}\ninperformEvaluation(src/tools/script.ts). Because0,\n${source}\nis not parenthesized, passing a multi-statement script with the defaultformat: "function"executes every statement after the first;in the page beforefn(...args)fails. - Append the underlying error message to the thrown
Errormessage inresolveScriptSource(src/tools/script.ts) instead of only setting{cause: error}.McpResponseonly formatserror.messageand ignoreserror.cause, which hides the root cause (ENOENT,EACCES, or root validation failure) from the tool output. - Remove the
as ParsedArgumentscast (tests/tools/script.test.ts) and// @ts-expect-errorcomment (src/tools/script.ts), addafterEach(() => sinon.restore()), and convert thesourcePathfile-loading tests intests/tools/script.test.tsfromwithMcpContext+fs.mkdtemptocreateHandlerMocks().AGENTS.mdprohibitsascasts and@ts-expect-error, requiressinon.restore()inafterEach, and mandates mock-based unit tests for handler delegation. - Let's extract this feature in to a separate PR: Replace the hardcoded
commandName === 'evaluate_script' && (argName === 'sourcePath' || argName === 'filePath')check insrc/bin/chrome-devtools.tswith metadata-driven path resolution fromverifyFilesSchemainscripts/generate-cli.ts/src/config/cli-options.ts. Other file-accepting CLI commands (take_screenshot,take_snapshot,upload_file, etc.) suffer from the same client-vs-daemoncwdmismatch and should be resolved uniformly.
|
@achideal could you please rebase? |
1a1db8a to
6363225
Compare
|
@OrKoN Rebased onto the latest main and resolved the conflicts in src/tools/script.ts and tests/tools/script.test.ts. I also updated the affected tests for the latest ConfigParser, ToolHandler constructor, and createTempDir requirements. The PR is mergeable again; targeted tests pass. The full suite passes except for the existing Windows-specific large full-page screenshot failure (Page.captureScreenshot: Page is too large). |
9cf47a4
Summary
evaluate_scriptwithsourcePathfor loading JavaScript from the MCP server filesystemformat: function | scriptwhile preserving existing function calls, output redirection, and CLI positional syntaxSupersedes #1772.
Fixes #1775.
Test plan
npm run gennpm run test tests/tools/script.test.ts tests/e2e/chrome-devtools-commands.test.tsnpm run test(feature-related tests pass; the local Windows run has unrelated existing failures: symlink creation requires additional OS privileges, one extension-page timeout, and the large full-page screenshot exceeds Chrome's limit)