Skip to content

feat: support evaluating scripts from local files - #2606

Merged
OrKoN merged 3 commits into
ChromeDevTools:mainfrom
achideal:feat/evaluate-script-source-path
Oct 1, 2026
Merged

OrKoN merged 3 commits into
ChromeDevTools:mainfrom
achideal:feat/evaluate-script-source-path

Conversation

@achideal

@achideal achideal commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • extend evaluate_script with sourcePath for loading JavaScript from the MCP server filesystem
  • add format: function | script while preserving existing function calls, output redirection, and CLI positional syntax
  • validate local source paths through MCP roots and cover page, service worker, and CLI execution

Supersedes #1772.
Fixes #1775.

Test plan

  • npm run gen
  • npm run test tests/tools/script.test.ts tests/e2e/chrome-devtools-commands.test.ts
  • npm run test (feature-related tests pass; the local Windows run has unrelated existing failures: symlink creation requires additional OS privileges, one extension-page timeout, and the large full-page screenshot exceeds Chrome's limit)

@google-cla

google-cla Bot commented Aug 22, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@achideal
achideal force-pushed the feat/evaluate-script-source-path branch 2 times, most recently from 2616476 to 945ef28 Compare August 26, 2026 04:26

@cbi58105-crypto cbi58105-crypto left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: Adds sourcePath + format to evaluate_script allowing loading local files; supports 'function' and 'script' modes; validates exactly one source and disallows args when format='script'. Tests added covering file loading and error cases. Suggestions: clarify how relative sourcePath is resolved (process.cwd vs repo root) and document file: URL handling; consider clearer error messages for unreadable paths; note ESM modules are not supported. I couldn't run tests locally due to environment (PowerShell execution policy / missing tsc).

@OrKoN
OrKoN self-requested a review September 16, 2026 06:00

@OrKoN OrKoN left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issues & Potential Bugs

1. format: "function" Wrapping Bug with Single-Line Comments and Trailing Semicolons

  • Location: src/tools/script.ts:328 (using fn = await evaluatable.evaluateHandle(\(${source})`);`)
  • Problem:
    1. Single-line comments at EOF: If a file loaded via sourcePath (or inline function) ends with a single-line comment without a trailing newline (e.g. () => document.title // get title), wrapping without newlines produces (() => document.title // get title). The closing parenthesis ) is commented out, throwing SyntaxError: Unexpected end of input. Wrapping with newlines ((\n${source}\n)) prevents this.
    2. Trailing semicolons in formatted .js files: Because format defaults to 'function' even when sourcePath is provided, loading a standard .js file formatted by Prettier/ESLint (which appends a trailing semicolon, e.g. () => { return 1; };) produces (() => { return 1; };), throwing SyntaxError: Unexpected token ';'.
  • Recommendation: Wrap with newlines ((\n${source}\n)), trim trailing semicolons/whitespace when format === 'function', or consider defaulting format to 'script' when sourcePath is used without an explicit format.

2. Serialization Discrepancy Between format: "function" and format: "script"

  • Location: src/tools/script.ts:327-340 (performEvaluation)
  • Problem:
    • In format: "function", JSON.stringify(await fn(...args)) executes inside the browser context.
    • In format: "script", const value = await evaluatable.evaluate(source) relies on CDP Runtime.evaluate({ returnByValue: true }) to transfer the value to Node.js, and then calls JSON.stringify(value) in Node.js.
    • CDP's C++ returnByValue serializer does not invoke .toJSON() and ignores prototype getters. Consequently, browser objects like DOMRect (el.getBoundingClientRect()), PerformanceEntry, or URL serialize completely in format: "function" ({"x":0,"y":0,"width":100,...}) but serialize to {} in format: "script".
    • Additionally, if a script evaluates to a non-serializable object (e.g. a DOM node or window), CDP throws a protocol error (Could not serialize object / Object reference chain is too long).

3. Relative sourcePath Resolution Bug in CLI Daemon Architecture

  • Location: src/bin/chrome-devtools.ts:288-302 & skills/chrome-devtools-cli/SKILL.md:63
  • Problem:
    • SKILL.md recommends running:
      chrome-devtools evaluate_script --pageId 1 --sourcePath ./script.js --format script
    • In src/bin/chrome-devtools.ts, the CLI client forwards commandArgs (including relative paths like ./script.js) verbatim over IPC to the background chrome-devtools-mcp daemon process.
    • If the background daemon was started in a different working directory than the user's current shell cwd, resolveScriptSource resolves ./script.js relative to the daemon's working directory rather than the CLI caller's working directory, resulting in Unable to read script source or reading the wrong file.
    • Note: The E2E test in tests/e2e/chrome-devtools-commands.test.ts:396 passes an absolute path (path.join(os.tmpdir(), 'script.js')), which masked this bug.
  • Recommendation: Resolve relative file paths (sourcePath, filePath) against process.cwd() in the CLI client (src/bin/chrome-devtools.ts) before forwarding the command payload to the daemon, or pass the client's cwd to the daemon.

Testing & Coverage Issues

4. Violation of AGENTS.md Testing Rules (withMcpContext Used for Unit Validation Tests)

  • Location: tests/tools/script.test.ts:511-562
  • Problem:
    • AGENTS.md specifies:

      Prefer mock-based unit tests over real-browser tests: Do not use withMcpContext or launch a real browser unless the test genuinely requires real browser or DevTools protocol integration...

    • The new tests requires exactly one script source, rejects args for classic scripts, and reports unreadable source files all launch a real Chrome browser via withMcpContext solely to assert synchronous/early parameter validation errors that throw before touching context, page, or Puppeteer.
  • Recommendation: Use createHandlerMocks() from tests/mocks.ts for handler validation unit tests.

5. Missing Test Coverage for verifyFilesSchema (sourcePath: true) and Relative CLI Paths

  • Problem:
    • All tests in tests/tools/script.test.ts call evaluateScript().handler(...) directly, bypassing ToolHandler.handle() where verifyFilesSchema (validateToolFiles in src/ToolHandler.ts) validates paths against context.validatePath(). There is no test verifying that sourcePath is checked by ToolHandler or that file:// URLs rewritten by ToolHandler are handled end-to-end.
    • tests/e2e/chrome-devtools-commands.test.ts only tests an absolute sourcePath; adding an E2E test with a relative --sourcePath ./script.js from a different working directory will catch the CLI/daemon CWD mismatch described in Issue #3.

Comment thread src/tools/script.ts Outdated
@achideal
achideal force-pushed the feat/evaluate-script-source-path branch from 945ef28 to 98e8169 Compare September 16, 2026 10:04
@achideal

Copy link
Copy Markdown
Contributor Author

@OrKoN I've addressed the review feedback and rebased the PR onto the latest main.

  • Function sources now handle trailing semicolons and single-line comments.
  • Script results are serialized in the browser context through a JSHandle, matching function mode.
  • Source files are loaded through McpContext.loadResource(), preserving path/root validation.
  • The CLI resolves relative sourcePath and filePath values before daemon IPC.
  • Synchronous validation tests now use handler mocks; ToolHandler, relative-path, and file-URL coverage were added.

Validation: npm run gen; targeted script, ToolHandler, CLI E2E, daemon, and pages tests pass. The full suite only retains the existing Windows-specific large full-page screenshot failure (Page.captureScreenshot: Page is too large). Please take another look when convenient.

@achideal
achideal force-pushed the feat/evaluate-script-source-path branch from 98e8169 to bda681f Compare September 22, 2026 06:15
@achideal
achideal requested a review from OrKoN September 22, 2026 07:11
@OrKoN
OrKoN force-pushed the feat/evaluate-script-source-path branch from bda681f to 27830bd Compare September 22, 2026 09:01

@OrKoN OrKoN left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a few more changes:

  • Keep source wrapped in parentheses ((\n${source}\n), stripping an optional trailing semicolon first) instead of 0,\n${source}\n in performEvaluation (src/tools/script.ts). Because 0,\n${source}\n is not parenthesized, passing a multi-statement script with the default format: "function" executes every statement after the first ; in the page before fn(...args) fails.
  • Append the underlying error message to the thrown Error message in resolveScriptSource (src/tools/script.ts) instead of only setting {cause: error}. McpResponse only formats error.message and ignores error.cause, which hides the root cause (ENOENT, EACCES, or root validation failure) from the tool output.
  • Remove the as ParsedArguments cast (tests/tools/script.test.ts) and // @ts-expect-error comment (src/tools/script.ts), add afterEach(() => sinon.restore()), and convert the sourcePath file-loading tests in tests/tools/script.test.ts from withMcpContext + fs.mkdtemp to createHandlerMocks(). AGENTS.md prohibits as casts and @ts-expect-error, requires sinon.restore() in afterEach, and mandates mock-based unit tests for handler delegation.
  • Let's extract this feature in to a separate PR: Replace the hardcoded commandName === 'evaluate_script' && (argName === 'sourcePath' || argName === 'filePath') check in src/bin/chrome-devtools.ts with metadata-driven path resolution from verifyFilesSchema in scripts/generate-cli.ts / src/config/cli-options.ts. Other file-accepting CLI commands (take_screenshot, take_snapshot, upload_file, etc.) suffer from the same client-vs-daemon cwd mismatch and should be resolved uniformly.

@OrKoN

OrKoN commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

@achideal could you please rebase?

@achideal
achideal force-pushed the feat/evaluate-script-source-path branch from 1a1db8a to 6363225 Compare September 30, 2026 14:35
@achideal

Copy link
Copy Markdown
Contributor Author

@OrKoN Rebased onto the latest main and resolved the conflicts in src/tools/script.ts and tests/tools/script.test.ts. I also updated the affected tests for the latest ConfigParser, ToolHandler constructor, and createTempDir requirements. The PR is mergeable again; targeted tests pass. The full suite passes except for the existing Windows-specific large full-page screenshot failure (Page.captureScreenshot: Page is too large).

@OrKoN
OrKoN enabled auto-merge September 30, 2026 14:44
@OrKoN
OrKoN added this pull request to the merge queue Oct 1, 2026
Merged via the queue into ChromeDevTools:main with commit 9cf47a4 Oct 1, 2026
35 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Add evaluate_script_file tool to evaluate JavaScript files from the local filesystem

3 participants