test: improve Chrome Extension coverage with CDP/MV3

[mxschmitt]

Motivation: This improves the test harness in preparation for #36459.

Whats included in this PR?

1. Move extensions tests out to extensions.spec.ts.
2. Split into MV2/MV3 tests
3. Copy existing MV2 tests over to MV3.
4. Rename actual extensions to contain MV2/MV3
5. Add a test for installing via CDP (last one)

Potential follow-ups:

1. Adjust our docs to recommend installing the extensions via CDP. Risk? It only works with MV3, not with MV2.
2. Remove MV2 tests. When? Lets keep it until its removed from Chrome Core.
3. Remove --disable-extensions flag from chromiumSwitches.ts. Risk? Unknown - using ignoreDefaultArgs might be fine for now.

For Chrome Beta this means:

1. We can't run MV2 tests anymore
2. We need to run all MV3 tests via CDP

Both I consider a follow-up as well.

[github-actions]

Test results for "tests 1"

7 flaky

⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-ubuntu-22.04-node18
⚠️ [firefox-page] › page/expect-misc.spec.ts:464:7 › toBeInViewport › should have good stack @firefox-ubuntu-22.04-node18
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104:3 › should work with strict CSP policy @firefox-ubuntu-22.04-node18
⚠️ [playwright-test] › ui-mode-trace.spec.ts:587:5 › should indicate current test status @macos-latest-node18-1
⚠️ [playwright-test] › ui-mode-test-watch.spec.ts:145:5 › should watch all @ubuntu-latest-node20-1
⚠️ [playwright-test] › ui-mode-test-watch.spec.ts:145:5 › should watch all @ubuntu-latest-node22-1
⚠️ [webkit-library] › library/browsercontext-add-cookies.spec.ts:426:3 › should allow unnamed cookies @webkit-ubuntu-22.04-node18

46457 passed, 931 skipped
✔️✔️✔️

Merge workflow run.

[github-actions]

Test results for "tests others"

111 failed
❌ [chromium-library] › library/browsercontext-cookies-third-party.spec.ts:526:1 › should(not) block third party cookies - persistent context @legacy-progress-timeouts-linux
❌ [chromium-library] › library/browsercontext-cookies-third-party.spec.ts:531:1 › should(not) block third party cookies - ephemeral context @legacy-progress-timeouts-linux
❌ [chromium-library] › library/browsercontext-har.spec.ts:529:3 › should ignore aborted requests @legacy-progress-timeouts-linux
❌ [chromium-library] › library/capabilities.spec.ts:94:3 › should play audio @smoke @legacy-progress-timeouts-linux
❌ [chromium-library] › library/capabilities.spec.ts:126:3 › should not crash on page with mp4 @smoke @legacy-progress-timeouts-linux
❌ [chromium-library] › library/capabilities.spec.ts:132:3 › should not crash on showDirectoryPicker @legacy-progress-timeouts-linux
❌ [chromium-library] › library/headful.spec.ts:122:3 › should(not) block third party cookies @legacy-progress-timeouts-linux
❌ [chromium-library] › library/headful.spec.ts:252:3 › should click bottom row w/ infobar in OOPIF @legacy-progress-timeouts-linux
❌ [chromium-library] › library/hit-target.spec.ts:24:3 › should block all events when hit target is wrong @legacy-progress-timeouts-linux
❌ [chromium-library] › library/hit-target.spec.ts:89:3 › should block all events when hit target is wrong and element detaches @legacy-progress-timeouts-linux
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:116:7 › cli codegen › should click after same-document navigation @legacy-progress-timeouts-linux
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:962:7 › cli codegen › should record omnibox navigations after performAction @legacy-progress-timeouts-linux
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:974:7 › cli codegen › should record omnibox navigations after recordAction @legacy-progress-timeouts-linux
❌ [chromium-library] › library/page-clock.spec.ts:327:3 › popup › should run time before popup @legacy-progress-timeouts-linux
❌ [chromium-library] › library/page-clock.spec.ts:343:3 › popup › should not run time before popup on pause @legacy-progress-timeouts-linux
❌ [chromium-library] › library/page-clock.spec.ts:400:3 › while running › should progress time @legacy-progress-timeouts-linux
❌ [chromium-library] › library/page-clock.spec.ts:436:3 › while running › should pause @legacy-progress-timeouts-linux
❌ [chromium-library] › library/page-clock.spec.ts:516:3 › Date.now › check Date.now is an integer @legacy-progress-timeouts-linux
❌ [chromium-library] › library/popup.spec.ts:222:3 › should not dispatch binding on a closed page @legacy-progress-timeouts-linux
❌ [chromium-library] › library/screenshot.spec.ts:289:14 › element screenshot › should restore viewport after page screenshot and timeout @legacy-progress-timeouts-linux
❌ [chromium-library] › library/snapshotter.spec.ts:199:3 › snapshots › should capture iframe with srcdoc @legacy-progress-timeouts-linux
❌ [chromium-library] › library/trace-viewer.spec.ts:92:1 › should show empty trace viewer @legacy-progress-timeouts-linux
❌ [chromium-library] › library/tracing.spec.ts:28:5 › should collect trace with resources, but no js @legacy-progress-timeouts-linux
❌ [chromium-library] › library/tracing.spec.ts:542:5 › should export trace concurrently to second navigation @legacy-progress-timeouts-linux
❌ [chromium-page] › page/elementhandle-scroll-into-view.spec.ts:58:3 › should wait for display:none to become visible @legacy-progress-timeouts-linux
❌ [chromium-page] › page/elementhandle-scroll-into-view.spec.ts💯3 › should wait for nested display:none to become visible @legacy-progress-timeouts-linux
❌ [chromium-page] › page/elementhandle-scroll-into-view.spec.ts:105:3 › should wait for element to stop moving @legacy-progress-timeouts-linux
❌ [chromium-page] › page/elementhandle-select-text.spec.ts:51:3 › should wait for visible @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:54:7 › toBeChecked › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:74:7 › toBeChecked › fail with not @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:83:7 › toBeChecked › fail with checked:false @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:92:7 › toBeChecked › fail with indeterminate: true @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts💯7 › toBeChecked › fail missing @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:351:7 › toBeVisible › with frameLocator @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:363:7 › toBeVisible › with frameLocator 2 @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:375:7 › toBeVisible › over navigation @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:438:7 › toBeHidden › fail with not when nothing matching @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:643:7 › toBeAttached › with frameLocator @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-boolean.spec.ts:655:7 › toBeAttached › over navigation @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-matcher-result.spec.ts:20:5 › toMatchText-based assertions should have matcher result @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-matcher-result.spec.ts:69:5 › toBeTruthy-based assertions should have matcher result @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-matcher-result.spec.ts:117:5 › toEqual-based assertions should have matcher result @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-matcher-result.spec.ts:164:5 › toBeChecked({ checked }) should have expected @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:22:7 › toHaveCount › toHaveCount pass @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:67:7 › toHaveCount › fail zero @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:75:7 › toHaveCount › fail zero 2 @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:92:7 › toHaveJSProperty › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:107:7 › toHaveJSProperty › fail string @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:123:7 › toHaveJSProperty › fail number @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:139:7 › toHaveJSProperty › fail boolean @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:155:7 › toHaveJSProperty › fail boolean 2 @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:187:7 › toHaveJSProperty › fail nested @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:210:7 › toHaveClass › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:224:7 › toHaveClass › fail with array @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:250:7 › toContainClass › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:267:7 › toContainClass › fail with array @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:282:7 › toHaveTitle › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:296:7 › toHaveURL › fail string @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:350:7 › toHaveAttribute › should not match missing attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:367:7 › toHaveAttribute › should match boolean attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-misc.spec.ts:464:7 › toBeInViewport › should have good stack @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-timeout.spec.ts:20:5 › should print timed out error message @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-timeout.spec.ts:26:5 › should print timed out error message when value does not match @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-timeout.spec.ts:32:5 › should print timed out error message with impossible timeout @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-timeout.spec.ts:38:5 › should print timed out error message when value does not match with impossible timeout @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-timeout.spec.ts:66:5 › should timeout during first locator handler check @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:34:7 › toHaveText with regex › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:53:7 › toContainText with regex › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:90:7 › toHaveText with text › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:98:7 › toHaveText with text › pass eventually @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:134:7 › toHaveText with text › fail with impossible timeout @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:151:7 › not.toHaveText › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:159:7 › not.toHaveText › should work when selector does not match @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:207:7 › toHaveText with array › fail on not+empty @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:224:7 › toHaveText with array › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-text.spec.ts:252:7 › toContainText with array › fail @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-value.spec.ts:76:7 › toHaveValues with multi-select › exact match with text failure @legacy-progress-timeouts-linux
❌ [chromium-page] › page/expect-to-have-value.spec.ts:103:7 › toHaveValues with multi-select › fails when items not selected @legacy-progress-timeouts-linux
❌ [chromium-page] › page/matchers.misc.spec.ts:28:3 › should print no-locator-resolved error when locator matcher did not resolve to any element @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-autowaiting-no-hang.spec.ts:69:3 › clicking in the middle of navigation that aborts @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-autowaiting-no-hang.spec.ts:93:3 › clicking in the middle of navigation that commits @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-autowaiting-no-hang.spec.ts:119:3 › goBack in the middle of navigation that commits @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-click-timeout-1.spec.ts:20:3 › should avoid side effects after timeout @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-click-timeout-4.spec.ts:36:3 › should click for the second time after first timeout @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-dispatchevent.spec.ts:87:3 › should dispatch click when node is added in shadow dom @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-event-load.spec.ts:20:3 › should fire once @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-event-load.spec.ts:31:3 › should fire once with iframe navigation @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-goto.spec.ts:518:3 › should succeed on url bar navigation when there is pending navigation @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-goto.spec.ts:790:3 › should not resolve goto upon window.stop() @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-listeners.spec.ts:23:3 › should not throw with ignoreErrors @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-network-idle.spec.ts:159:3 › should wait for networkidle when iframe attaches and detaches @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-request-continue.spec.ts:826:3 › should intercept css variable with background url @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-set-content.spec.ts:111:3 › content() should throw nice error during navigation @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-wait-for-function.spec.ts:21:3 › should timeout @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-wait-for-function.spec.ts:55:3 › should avoid side effects after timeout @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-wait-for-selector-1.spec.ts:183:3 › should resolve promise when node is added in shadow dom @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-wait-for-selector-2.spec.ts:332:3 › should fail if element handle was detached while waiting @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-wait-for-selector-2.spec.ts:342:3 › should succeed if element handle was detached while waiting for hidden @legacy-progress-timeouts-linux
❌ [chromium-page] › page/page-wait-for-selector-2.spec.ts:351:3 › should succeed if element handle was detached while waiting for detached @legacy-progress-timeouts-linux
❌ [chromium-page] › page/selectors-frame.spec.ts:302:3 › click should survive navigation @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:130:5 › checked attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:169:5 › disabled attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:201:5 › expanded attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:233:5 › level attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:261:5 › pressed attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:309:5 › selected attribute @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:404:5 › expected formatter @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:674:5 › should not match what is not matched @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:697:5 › should detect unexpected children: equal @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:723:5 › should detect unexpected children: deep-equal @legacy-progress-timeouts-linux
❌ [chromium-page] › page/to-match-aria-snapshot.spec.ts:762:5 › should allow restoring contain mode inside deep-equal @legacy-progress-timeouts-linux

3 flaky

⚠️ [chromium-library] › library/snapshotter.spec.ts:149:3 › snapshots › should capture frame @legacy-progress-timeouts-linux
⚠️ [chromium-library] › library/snapshotter.spec.ts:173:3 › snapshots › should capture iframe @legacy-progress-timeouts-linux
⚠️ [playwright-test] › ui-mode-test-watch.spec.ts:145:5 › should watch all @realtime-time-runner-chromium-linux

26056 passed, 595 skipped, 82 did not run
✔️✔️✔️

Merge workflow run.

[github-actions]

Test results for "tests 2"

5 fatal errors, not part of any test
59 failed
❌ [chromium-library] › library/inspector/title.spec.ts:35:5 › should update primary page URL when original primary closes @channel-chromium-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @channel-chromium-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @channel-chromium-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @channel-chromium-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:23:5 › MV2 › should return background pages @chrome-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:44:5 › MV2 › should return background pages when recording video @chrome-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:66:5 › MV2 › should support request/response events when using backgroundPage() @chrome-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chrome-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chrome-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @chrome-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:192:5 › MV3 › should report console messages from content script via CLI @chrome-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:23:5 › MV2 › should return background pages @chrome-beta-ubuntu-22.04
❌ [chromium-library] › library/chromium/extensions.spec.ts:44:5 › MV2 › should return background pages when recording video @chrome-beta-ubuntu-22.04
❌ [chromium-library] › library/chromium/extensions.spec.ts:66:5 › MV2 › should support request/response events when using backgroundPage() @chrome-beta-ubuntu-22.04
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chrome-beta-ubuntu-22.04
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chrome-beta-ubuntu-22.04
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @chrome-beta-ubuntu-22.04
❌ [chromium-library] › library/chromium/extensions.spec.ts:192:5 › MV3 › should report console messages from content script via CLI @chrome-beta-ubuntu-22.04
❌ [chromium-library] › library/chromium/extensions.spec.ts:23:5 › MV2 › should return background pages @chrome-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:44:5 › MV2 › should return background pages when recording video @chrome-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:66:5 › MV2 › should support request/response events when using backgroundPage() @chrome-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chrome-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chrome-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @chrome-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:192:5 › MV3 › should report console messages from content script via CLI @chrome-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chrome-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chrome-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @chrome-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chromium-headed-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chromium-headed-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @chromium-headed-windows-latest
❌ [chromium-library] › library/capabilities.spec.ts:94:3 › should play audio @smoke @chromium-macos-14-large
❌ [chromium-library] › library/chromium/tracing.spec.ts:49:3 › should run with custom categories if provided @chromium-tip-of-tree-macos-13--headed
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chromium-tip-of-tree-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chromium-tip-of-tree-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @chromium-tip-of-tree-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chromium-tip-of-tree-windows-latest--headed
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @chromium-tip-of-tree-windows-latest--headed
❌ [firefox-page] › page/expect-misc.spec.ts:464:7 › toBeInViewport › should have good stack @firefox-headed-ubuntu-24.04
❌ [firefox-library] › library/inspector/cli-codegen-3.spec.ts:667:7 › cli codegen › should consume contextmenu events, despite a custom context menu @firefox-macos-13-large
❌ [firefox-library] › library/capabilities.spec.ts:94:3 › should play audio @smoke @firefox-macos-14-large
❌ [chromium-library] › library/proxy.spec.ts:93:11 › should proxy local network requests › by default › loopback address @msedge-beta-macos-latest
❌ [chromium-library] › library/proxy.spec.ts:93:11 › should proxy local network requests › by default › link-local @msedge-beta-macos-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @msedge-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @msedge-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @msedge-beta-windows-latest
❌ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @msedge-dev-windows-latest
❌ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-macos-latest
❌ [webkit-library] › library/browsercontext-pages.spec.ts:82:3 › should click the button with offset with page scale @tracing-webkit
❌ [webkit-library] › library/browsercontext-pages.spec.ts:105:3 › should return bounding box with page scale @tracing-webkit
❌ [webkit-library] › library/multiclient.spec.ts:107:1 › should receive viewport size changes @tracing-webkit
❌ [webkit-library] › library/capabilities.spec.ts:67:3 › should play video @smoke @webkit-macos-14-large
❌ [webkit-library] › library/capabilities.spec.ts:94:3 › should play audio @smoke @webkit-macos-14-large
❌ [webkit-library] › library/capabilities.spec.ts:126:3 › should not crash on page with mp4 @smoke @webkit-macos-14-large
❌ [webkit-library] › library/capabilities.spec.ts:67:3 › should play video @smoke @webkit-macos-15-large
❌ [webkit-library] › library/capabilities.spec.ts:94:3 › should play audio @smoke @webkit-macos-15-large
❌ [webkit-library] › library/capabilities.spec.ts:126:3 › should not crash on page with mp4 @smoke @webkit-macos-15-large
❌ [webkit-library] › library/tracing.spec.ts:430:14 › should produce screencast frames fit @webkit-macos-15-xlarge
❌ [webkit-library] › library/tracing.spec.ts:430:14 › should produce screencast frames crop @webkit-macos-15-xlarge

148 flaky

⚠️ [chromium-library] › library/beforeunload.spec.ts:20:3 › should close browser with beforeunload page @channel-chromium-macos-latest
⚠️ [chromium-library] › library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @channel-chromium-macos-latest
⚠️ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @channel-chromium-macos-latest
⚠️ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @channel-chromium-macos-latest
⚠️ [chromium-library] › library/chromium/launcher.spec.ts:28:3 › should not throw with remote-debugging-port argument @channel-chromium-macos-latest
⚠️ [chromium-library] › library/debug-controller.spec.ts:153:1 › should highlight all @channel-chromium-macos-latest
⚠️ [chromium-library] › library/defaultbrowsercontext-1.spec.ts:28:3 › context.cookies() should work @smoke @channel-chromium-macos-latest
⚠️ [chromium-library] › library/inspector/cli-codegen-2.spec.ts:93:7 › cli codegen › should not lead to an error if html gets clicked @channel-chromium-macos-latest
⚠️ [chromium-library] › library/inspector/pause.spec.ts:23:3 › should resume when closing inspector @channel-chromium-macos-latest
⚠️ [chromium-library] › library/page-close.spec.ts:162:5 › should not treat navigations as new popups @channel-chromium-macos-latest
⚠️ [chromium-library] › library/trace-viewer.spec.ts:1564:1 › should not leak recorders @channel-chromium-macos-latest
⚠️ [chromium-library] › library/video.spec.ts:163:5 › screencast › should work with old options @channel-chromium-macos-latest
⚠️ [chromium-page] › page/workers.spec.ts:271:3 › should support extra http headers @channel-chromium-macos-latest
⚠️ [chromium-library] › library/video.spec.ts:580:5 › screencast › should capture static page in persistent context @smoke @channel-chromium-windows-latest
⚠️ [chromium-library] › library/browser.spec.ts:54:5 › should dispatch page.on(close) upon browser.close and reject evaluate @chrome-beta-macos-latest
⚠️ [chromium-library] › library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @chrome-beta-macos-latest
⚠️ [chromium-library] › library/browsertype-launch-server.spec.ts:23:5 › launch server › should work @chrome-beta-macos-latest
⚠️ [chromium-library] › library/chromium/tracing.spec.ts:98:3 › should support a buffer without a path @chrome-beta-macos-latest
⚠️ [chromium-library] › library/defaultbrowsercontext-2.spec.ts:191:3 › should fire close event for a persistent context @chrome-beta-macos-latest
⚠️ [chromium-page] › page/workers.spec.ts:150:3 › should report network activity @chrome-beta-macos-latest
⚠️ [chromium-page] › page/workers.spec.ts:271:3 › should support extra http headers @chrome-beta-macos-latest
⚠️ [chromium-library] › library/debug-controller.spec.ts:71:1 › should pick element @chrome-beta-ubuntu-22.04
⚠️ [chromium-library] › library/debug-controller.spec.ts:71:1 › should pick element @chrome-beta-windows-latest
⚠️ [chromium-library] › library/browser.spec.ts:54:5 › should dispatch page.on(close) upon browser.close and reject evaluate @chrome-macos-latest
⚠️ [chromium-library] › library/browsertype-launch-server.spec.ts:23:5 › launch server › should work @chrome-macos-latest
⚠️ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chrome-macos-latest
⚠️ [chromium-library] › library/client-certificates.spec.ts:705:3 › browser › support http2 if the browser only supports http1.1 @chrome-macos-latest
⚠️ [chromium-library] › library/debug-controller.spec.ts:130:1 › should navigate all @chrome-macos-latest
⚠️ [chromium-library] › library/debug-controller.spec.ts:153:1 › should highlight all @chrome-macos-latest
⚠️ [chromium-library] › library/defaultbrowsercontext-1.spec.ts:28:3 › context.cookies() should work @smoke @chrome-macos-latest
⚠️ [chromium-library] › library/inspector/cli-codegen-2.spec.ts:93:7 › cli codegen › should not lead to an error if html gets clicked @chrome-macos-latest
⚠️ [chromium-library] › library/inspector/pause.spec.ts:23:3 › should resume when closing inspector @chrome-macos-latest
⚠️ [chromium-page] › page/workers.spec.ts:271:3 › should support extra http headers @chrome-macos-latest
⚠️ [chromium-library] › library/selector-generator.spec.ts:582:5 › selector generator › should generate relative selector @chrome-windows-latest
⚠️ [chromium-library] › library/video.spec.ts:187:5 › screencast › should capture static page @chromium-headed-macos-14-xlarge
⚠️ [chromium-library] › library/inspector/pause.spec.ts:529:5 › pause › should record from debugger @chromium-headed-ubuntu-24.04
⚠️ [chromium-library] › library/inspector/cli-codegen-pick-locator.spec.ts:35:7 › should update locator highlight @chromium-headed-windows-latest
⚠️ [chromium-page] › page/expect-misc.spec.ts:464:7 › toBeInViewport › should have good stack @chromium-macos-14-large
⚠️ [chromium-library] › library/chromium/tracing.spec.ts:49:3 › should run with custom categories if provided @chromium-tip-of-tree-headless-shell-ubuntu-22.04
⚠️ [chromium-library] › library/chromium/extensions.spec.ts:131:5 › MV3 › should return background pages when recording video @chromium-tip-of-tree-macos-13--headed
⚠️ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @chromium-tip-of-tree-windows-latest--headed
⚠️ [firefox-library] › library/browsertype-launch.spec.ts:89:3 › should handle exception @firefox-beta-macos-latest
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-beta-macos-latest
⚠️ [firefox-page] › page/page-add-script-tag.spec.ts:93:3 › should throw when added with content to the CSP page @firefox-beta-macos-latest
⚠️ [firefox-page] › page/page-emulate-media.spec.ts:144:3 › should keep reduced motion and color emulation after reload @firefox-beta-macos-latest
⚠️ [firefox-page] › page/page-event-request.spec.ts:182:3 › should return response body when Cross-Origin-Opener-Policy is set @firefox-beta-macos-latest
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-beta-ubuntu-22.04
⚠️ [firefox-page] › page/page-goto.spec.ts:81:3 › should work with Cross-Origin-Opener-Policy @firefox-beta-ubuntu-22.04
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-beta-windows-latest
⚠️ [firefox-library] › library/inspector/cli-codegen-3.spec.ts:222:7 › cli codegen › should generate frame locators (4) @firefox-beta-windows-latest
⚠️ [firefox-library] › library/capabilities.spec.ts:252:3 › requestFullscreen @firefox-headed-macos-14-xlarge
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-headed-macos-14-xlarge
⚠️ [firefox-library] › library/browsercontext-basic.spec.ts:36:3 › should be able to click across browser contexts @firefox-headed-ubuntu-24.04
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-headed-ubuntu-24.04
⚠️ [firefox-page] › page/page-add-locator-handler.spec.ts:212:5 › should work when owner frame detaches @firefox-headed-ubuntu-24.04
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104:3 › should work with strict CSP policy @firefox-headed-ubuntu-24.04
⚠️ [firefox-library] › library/browsercontext-basic.spec.ts:36:3 › should be able to click across browser contexts @firefox-headed-windows-latest
⚠️ [firefox-library] › library/browsercontext-fetch.spec.ts:293:3 › should add cookies from Set-Cookie header @firefox-headed-windows-latest
⚠️ [firefox-library] › library/browsercontext-proxy.spec.ts:55:3 › should use proxy @firefox-headed-windows-latest
⚠️ [firefox-library] › library/capabilities.spec.ts:126:3 › should not crash on page with mp4 @smoke @firefox-headed-windows-latest
⚠️ [firefox-library] › library/client-certificates.spec.ts:652:3 › browser › should pass with matching certificates and trailing slash @firefox-headed-windows-latest
⚠️ [firefox-library] › library/emulation-focus.spec.ts:65:3 › should not affect mouse event target page @firefox-headed-windows-latest
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-headed-windows-latest
⚠️ [firefox-library] › library/locator-dispatchevent-touch.spec.ts:21:3 › should support touch points in touch event arguments @firefox-headed-windows-latest
⚠️ [firefox-library] › library/page-clock.spec.ts:129:3 › runFor › passes 8 seconds @firefox-headed-windows-latest
⚠️ [firefox-library] › library/page-close.spec.ts:66:5 › should reject all promises when page is closed @firefox-headed-windows-latest
⚠️ [firefox-library] › library/video.spec.ts:379:5 › screencast › should capture navigation @firefox-headed-windows-latest
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104:3 › should work with strict CSP policy @firefox-headed-windows-latest
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-macos-13-large
⚠️ [firefox-library] › library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-13-large
⚠️ [firefox-library] › library/video.spec.ts:379:5 › screencast › should capture navigation @firefox-macos-13-large
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104:3 › should work with strict CSP policy @firefox-macos-13-large
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-macos-13-xlarge
⚠️ [firefox-library] › library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-13-xlarge
⚠️ [firefox-library] › library/capabilities.spec.ts:82:3 › should play webm video @smoke @firefox-macos-14-large
⚠️ [firefox-library] › library/capabilities.spec.ts:484:3 › should not crash on feature detection for PublicKeyCredential @firefox-macos-14-large
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-macos-14-large
⚠️ [firefox-library] › library/inspector/cli-codegen-pick-locator.spec.ts:35:7 › should update locator highlight @firefox-macos-14-large
⚠️ [firefox-library] › library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-14-large
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104:3 › should work with strict CSP policy @firefox-macos-14-large
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-macos-14-xlarge
⚠️ [firefox-library] › library/video.spec.ts:163:5 › screencast › should work with old options @firefox-macos-14-xlarge
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-ubuntu-24.04
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104:3 › should work with strict CSP policy @firefox-ubuntu-24.04
⚠️ [firefox-page] › page/workers.spec.ts:106:3 › should clear upon navigation @firefox-ubuntu-24.04
⚠️ [firefox-library] › library/debug-controller.spec.ts:71:1 › should pick element @firefox-windows-latest
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:986:7 › cli codegen › should not throw csp directive violation errors @firefox-windows-latest
⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104:3 › should work with strict CSP policy @firefox-windows-latest
⚠️ [chromium-library] › library/beforeunload.spec.ts:20:3 › should close browser with beforeunload page @msedge-beta-macos-latest
⚠️ [chromium-library] › library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @msedge-beta-macos-latest
⚠️ [chromium-library] › library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-beta-macos-latest
⚠️ [chromium-library] › library/browsertype-launch-server.spec.ts:47:5 › launch server › should work with wsPath @msedge-beta-macos-latest
⚠️ [chromium-library] › library/chromium/connect-over-cdp.spec.ts:90:5 › should connectOverCDP and manage downloads in default context @msedge-beta-macos-latest
⚠️ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-beta-macos-latest
⚠️ [chromium-library] › library/proxy.spec.ts:31:3 › should use proxy @smoke @msedge-beta-macos-latest
⚠️ [chromium-library] › library/proxy.spec.ts:93:11 › should proxy local network requests › with other bypasses › localhost @msedge-beta-macos-latest
⚠️ [chromium-page] › page/workers.spec.ts:25:3 › Page.workers @smoke @msedge-beta-macos-latest
⚠️ [chromium-library] › library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-beta-ubuntu-22.04
⚠️ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-beta-ubuntu-22.04
⚠️ [chromium-library] › library/browsercontext-proxy.spec.ts:103:3 › should set cookie for top-level domain @msedge-beta-windows-latest
⚠️ [chromium-library] › library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-beta-windows-latest
⚠️ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-beta-windows-latest
⚠️ [chromium-library] › library/video.spec.ts:580:5 › screencast › should capture static page in persistent context @smoke @msedge-beta-windows-latest
⚠️ [chromium-library] › library/beforeunload.spec.ts:20:3 › should close browser with beforeunload page @msedge-dev-macos-latest
⚠️ [chromium-library] › library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-dev-macos-latest
⚠️ [chromium-library] › library/debug-controller.spec.ts:141:1 › should reset for reuse @msedge-dev-macos-latest
⚠️ [chromium-library] › library/defaultbrowsercontext-2.spec.ts:156:3 › should have passed URL when launching with ignoreDefaultArgs: true @msedge-dev-macos-latest
⚠️ [chromium-library] › library/downloads-path.spec.ts:77:5 › downloads path › should report downloads in downloadsPath folder with a relative path @msedge-dev-macos-latest
⚠️ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-dev-macos-latest
⚠️ [chromium-library] › library/proxy.spec.ts:93:11 › should proxy local network requests › with other bypasses › link-local @msedge-dev-macos-latest
⚠️ [chromium-page] › page/workers.spec.ts:150:3 › should report network activity @msedge-dev-macos-latest
⚠️ [chromium-library] › library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-dev-ubuntu-22.04
⚠️ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-dev-ubuntu-22.04
⚠️ [chromium-library] › library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-dev-windows-latest
⚠️ [chromium-library] › library/chromium/extensions.spec.ts:113:5 › MV3 › should return background pages @msedge-dev-windows-latest
⚠️ [chromium-library] › library/chromium/extensions.spec.ts:151:5 › MV3 › should support request/response events when using backgroundPage() @msedge-dev-windows-latest
⚠️ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-dev-windows-latest
⚠️ [chromium-library] › library/video.spec.ts:580:5 › screencast › should capture static page in persistent context @smoke @msedge-dev-windows-latest
⚠️ [chromium-library] › library/beforeunload.spec.ts:20:3 › should close browser with beforeunload page @msedge-macos-latest
⚠️ [chromium-library] › library/browsercontext-base-url.spec.ts:37:3 › should construct a new URL when a baseURL in browserType.launchPersistentContext is passed to page.goto @msedge-macos-latest
⚠️ [chromium-library] › library/browsercontext-fetch.spec.ts:1262:3 › should work with connectOverCDP @msedge-macos-latest
⚠️ [chromium-library] › library/browsercontext-proxy.spec.ts:27:3 › should work when passing the proxy only on the context level @msedge-macos-latest
⚠️ [chromium-library] › library/browsertype-launch-server.spec.ts:23:5 › launch server › should work @msedge-macos-latest
⚠️ [chromium-library] › library/debug-controller.spec.ts:141:1 › should reset for reuse @msedge-macos-latest
⚠️ [chromium-library] › library/browsercontext-reuse.spec.ts:30:1 › should re-add binding after reset @msedge-ubuntu-22.04
⚠️ [chromium-library] › library/permissions.spec.ts:244:5 › should be able to use the local-fonts API @msedge-ubuntu-22.04
⚠️ [webkit-library] › library/multiclient.spec.ts:132:1 › should not allow parallel css coverage @tracing-webkit
⚠️ [webkit-library] › library/screenshot.spec.ts:276:14 › element screenshot › should restore viewport after page screenshot and exception @tracing-webkit
⚠️ [webkit-library] › library/video.spec.ts:411:5 › screencast › should capture css transformation @tracing-webkit
⚠️ [webkit-library] › library/video.spec.ts:441:5 › screencast › should work for popups @tracing-webkit
⚠️ [webkit-library] › library/inspector/cli-codegen-1.spec.ts:610:7 › cli codegen › should select @webkit-headed-macos-14-xlarge
⚠️ [webkit-library] › library/tracing.spec.ts:430:14 › should produce screencast frames scale @webkit-headed-macos-14-xlarge
⚠️ [webkit-library] › library/inspector/cli-codegen-1.spec.ts:559:7 › cli codegen › should check with keyboard @webkit-headed-ubuntu-22.04
⚠️ [webkit-library] › library/inspector/cli-codegen-3.spec.ts:636:7 › cli codegen › should consume pointer events @webkit-headed-ubuntu-22.04
⚠️ [webkit-page] › page/page-click.spec.ts:251:3 › should click on checkbox input and toggle @webkit-headed-ubuntu-24.04
⚠️ [webkit-page] › page/page-click.spec.ts:251:3 › should click on checkbox input and toggle @webkit-headed-windows-latest
⚠️ [webkit-library] › library/tracing.spec.ts:430:14 › should produce screencast frames fit @webkit-macos-13-xlarge
⚠️ [webkit-library] › library/tracing.spec.ts:430:14 › should produce screencast frames scale @webkit-macos-13-xlarge
⚠️ [webkit-library] › library/debug-controller.spec.ts:71:1 › should pick element @webkit-macos-14-large
⚠️ [webkit-library] › library/inspector/cli-codegen-3.spec.ts:222:7 › cli codegen › should generate frame locators (4) @webkit-macos-14-large
⚠️ [webkit-library] › library/tracing.spec.ts:430:14 › should produce screencast frames crop @webkit-macos-14-xlarge
⚠️ [webkit-library] › library/tracing.spec.ts:430:14 › should produce screencast frames scale @webkit-macos-14-xlarge
⚠️ [webkit-page] › page/page-request-continue.spec.ts:191:5 › post data › should amend post data @webkit-macos-14-xlarge
⚠️ [webkit-page] › page/wheel.spec.ts:51:3 › should dispatch wheel events @smoke @webkit-macos-14-xlarge
⚠️ [webkit-library] › library/debug-controller.spec.ts:71:1 › should pick element @webkit-macos-15-large
⚠️ [webkit-library] › library/inspector/cli-codegen-3.spec.ts:222:7 › cli codegen › should generate frame locators (4) @webkit-macos-15-large
⚠️ [webkit-page] › page/wheel.spec.ts:101:3 › should dispatch wheel events after popup was opened @smoke @webkit-macos-15-large
⚠️ [webkit-library] › library/inspector/cli-codegen-javascript.spec.ts:87:5 › should save the codegen output to a file if specified @webkit-windows-latest

245760 passed, 9749 skipped
✔️✔️✔️

Merge workflow run.