Skip to content

[feat]: add position to locator.click() - #3112

Open
antonvishal wants to merge 5 commits into
browserbase:mainfrom
antonvishal:feat/locator-click-position
Open

antonvishal wants to merge 5 commits into
browserbase:mainfrom
antonvishal:feat/locator-click-position

Conversation

@antonvishal

@antonvishal antonvishal commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

why

I was navigating across https://teleoperator.mindblown.ai/?mb=game and needed to click specific points on a canvas map. Stagehand's locator.click() could only click an element's center, and passing Playwright's position option was rejected by the protocol schema, so agents written against the Playwright facade failed there.

what changed

  • locator.click({ position: { x, y } }) clicks at a point measured from the top-left of the element's padding box. That matches Playwright and MouseEvent.offsetX/Y, so {x: 0, y: 0} lands exactly on the padding edge even when the element has padding and a border.
  • The point follows the element's box, so rotated and transformed elements work.
  • Added to the protocol schema (LocatorClickPosition), the extension locator and frame/deep locator delegates, and the Playwright facade (locator.click and frame locators, with input validation). The generated Go and Python models are regenerated, Python's Locator.click gains position=, and the docs are updated.
  • In the facade, force: true combined with position dispatches a real click at the point rather than the DOM-click shortcut.

test plan

  • New clickCount.test.ts integration test: page-reported offsets equal the requested position for a padded and bordered element ({0,0} and {37,19}) and for a rotated element. The old protocol schema rejects the option.
  • facade-frames.test.ts: positioned clicks land exactly on canvases in the main frame and in same-origin and out-of-process iframes.
  • Unit tests: the extension's click maths, the protocol schema, Python click(position=...), and Go LocatorClickOptions.Position.

Summary by cubic

Adds a position option to locator.click() so a click can target a point relative to the element's padding box instead of only its center, matching Playwright so Playwright-facade agents can interact with canvas and map UIs.

  • Coordinates are measured from the padding box's top-left corner, aligning with MouseEvent.offsetX/Y; {0,0} lands on the padding edge even with padding and borders, and the point follows rotated or transformed elements.
  • Exposed in the protocol schema, extension locators, and the Playwright facade (with input validation); in the facade, force: true combined with position dispatches a real click at the point.
  • Adds position to the Python and Go SDKs, regenerates generated models, and updates the docs.
  • Tests cover padding-box and rotated positioning in the main frame and in same-origin and out-of-process iframes.

Written for commit 253a3cb. Summary will update on new commits.

View guided diff

Click a point relative to the element's padding box, matching Playwright and
MouseEvent.offsetX/Y, so canvas and map UIs can be targeted through a locator.
Exposed in the protocol, TypeScript, Python, Go, and the Playwright facade.

Co-authored-by: Cursor <cursoragent@cursor.com>
@changeset-bot

changeset-bot Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 253a3cb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 25 packages
Name Type
@browserbasehq/stagehand-protocol Minor
@browserbasehq/stagehand-extension Minor
@browserbasehq/stagehand Minor
@browserbasehq/stagehand-python Minor
@browserbasehq/stagehand-go Minor
@browserbasehq/stagehand-integrations Minor
browse Patch
@browserbasehq/stagehand-examples Patch
@browserbasehq/stagehand-integrations-claude-cua-sdk Patch
@browserbasehq/eve Patch
@browserbasehq/stagehand-integrations-gemini-cua-sdk Patch
@browserbasehq/stagehand-integrations-example-pi-facade Patch
@browserbasehq/stagehand-integrations-claude-agent-sdk Patch
@browserbasehq/stagehand-integrations-example-claude-code-facade Patch
@browserbasehq/stagehand-integrations-codex-sdk Patch
@browserbasehq/stagehand-integrations-example-codex-facade Patch
@browserbasehq/stagehand-integrations-cursor-sdk Patch
@browserbasehq/stagehand-integrations-example-cursor-facade Patch
@browserbasehq/stagehand-integrations-deepagents-sdk Patch
@browserbasehq/stagehand-integrations-eve-sdk Patch
@browserbasehq/stagehand-integrations-fx-sdk Patch
@browserbasehq/stagehand-integrations-mastra-sdk Patch
@browserbasehq/stagehand-integrations-example-mastra-facade Patch
@browserbasehq/stagehand-integrations-pi-sdk Patch
@browserbasehq/stagehand-integrations-example-vercel-ai-facade Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run.
Approving the latest commit mirrors it into an internal PR owned by the approver.
If new commits are pushed later, the internal PR stays open but is marked stale until someone approves the latest external commit and refreshes it.

@github-actions github-actions Bot added external-contributor Tracks PRs mirrored from external contributor forks. external-contributor:awaiting-approval Waiting for a stagehand team member to approve the latest external commit. labels Oct 5, 2026
@antonvishal
antonvishal marked this pull request as ready for review October 5, 2026 19:02
@antonvishal
antonvishal requested a review from a team as a code owner October 5, 2026 19:02
# Conflicts:
#	packages/sdk-go/internal/extensionassets/stagehand-extension.zip
# Conflicts:
#	packages/sdk-go/internal/extensionassets/stagehand-extension.zip
# Conflicts:
#	packages/sdk-go/internal/extensionassets/stagehand-extension.zip
# Conflicts:
#	packages/sdk-go/internal/extensionassets/stagehand-extension.zip

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contributor:awaiting-approval Waiting for a stagehand team member to approve the latest external commit. external-contributor Tracks PRs mirrored from external contributor forks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant