Tool to validate locators

[ruifigueira]

Claude Sonnet can generate very accurate locators but there's no tool for validating them non destructively, so it cannot check if they exactly match the corresponding ref element.

I propose a new testing tool, validateLocator that would receive a locator and a ref and ensure that they match.

I did try it (I copied some code for parsing locators from playwright-core):

https://github.com/ruifigueira/playwright-mcp/blob/validate-locator-tool/src/tools/testing.ts#L67-L117

And results with Claude 3.5 Sonnet are quite good.

For instance, I created some todo entries in https://demo.playwright.dev/todomvc. with some repeated (in my case, "Hello, World!").

An example of a prompt I did:

User: In the current page, I want a locator for a todo entry checkbox that has "Hello, World!" as text

Agent: Let me help you create a locator for a todo entry checkbox with the text "Hello, World!". I'll use the browser_snapshot tool first to get the current page structure, and then I can help you with the most appropriate locator.

🛠️ Ran Page snapshot

I notice there are multiple todo entries in the list, including two with the text "Hello, World!". The best way to locate the checkbox for a specific todo entry with that text would be to use a locator that leverages both the checkbox and its associated text. Let me validate the most appropriate locator.

🛠️ Ran Validate locator

Input:

{
  "locator": "getByRole('checkbox', { name: 'Toggle Todo' }).filter({ has: getByText('Hello, World!', { exact: true }) })",
  "element": "checkbox for todo entry with text \"Hello, World!\"",
  "ref": "e15"
}

Output

Locator does not match any elements

Let me try a slightly different approach:

🛠️ Ran Validate locator
Input:

{
  "locator": "getByRole('listitem').filter({ hasText: 'Hello, World!' }).getByRole('checkbox')",
  "ref": "e15",
  "element": "checkbox for todo entry with text \"Hello, World!\""
}

Output

Locator is ambiguous, it matches the reference element but also other elements

Since there are multiple todo items with the same text, I'll need to be more specific. Let me show you how to handle this situation:

1. If you want to select the first "Hello, World!" todo checkbox:

getByRole('listitem').filter({ hasText: 'Hello, World!' }).first().getByRole('checkbox')

2. If you want to select a specific "Hello, World!" todo checkbox, you can use the nth item:

getByRole('listitem').filter({ hasText: 'Hello, World!' }).nth(0).getByRole('checkbox')  // First one
getByRole('listitem').filter({ hasText: 'Hello, World!' }).nth(1).getByRole('checkbox')  // Second one

Let me validate the first version:

🛠️ Ran Validate locator
Input:

{
  "locator": "getByRole('listitem').filter({ hasText: 'Hello, World!' }).first().getByRole('checkbox')",
  "ref": "e15",
  "element": "first checkbox for todo entry with text \"Hello, World!\""
}

Output

Locator is valid

The locator has been validated and works correctly. You can use:

getByRole('listitem').filter({ hasText: 'Hello, World!' }).first().getByRole('checkbox')

This locator will:

• Find all list items (getByRole('listitem'))
• Filter them to only those containing the text "Hello, World!" (filter({ hasText: 'Hello, World!' }))
• Take the first matching item (first())
• Get the checkbox within that item (getByRole('checkbox'))

If you need to interact with a specific "Hello, World!" todo item (like the second one), you can replace .first() with .nth(1) to get the second item instead.

[pavelfeldman]

I wonder if a noop tool (say hover) would get your there faster. Tools return locators for the elements they interacted with.

[ruifigueira]

I know, and most of the times it works well. But there are scenarios where it is just not enough.

One example: I disabled my Validate locator tool and asked the following:

use the provided tools to generate a playwright test:

• navigate to http://demo.playwright.dev/todomvc/
• create the following todo items:
  • travel to Paris
  • learn how to play piano
  • travel to Paris
• check the second one with "travel to Paris"
• mark as completed

And for the checkbox, it generates:

// Click checkbox "Toggle Todo" for the second "travel to Paris" item
await page.getByRole('checkbox', { name: 'Toggle Todo' }).nth(1).click();

which is not a good selector because it doesn't include the associated text, it just picks the second item, whatever its text is.

If the agent has the ability to be "creative" and generate its own locators and then validate them, at least there's a way to improve that test in ways that right now is not possible.

With the new tool, it was able to generate a correct locator:

await page.getByRole('listitem').filter({ hasText: 'travel to Paris' }).nth(1).getByRole('checkbox').click();