Implement test sharding in CI with workflow optimization to reduce build time #481

Copilot · 2025-08-29T06:38:58Z

This PR implements test sharding for Playwright tests across multiple OS platforms to significantly reduce CI build time. The implementation includes:

Key Features

Test Sharding: Splits tests into 4 shards across 3 operating systems (Ubuntu, Windows, macOS)
Workflow Optimization: Added a common Setup job that prepares the environment once and shares cached dependencies across all test matrix jobs
Enhanced Caching: Implements aggressive caching for both pnpm dependencies and Playwright browsers per OS
Report Merging: Combines test reports from all shards into a single comprehensive report

Technical Implementation

Added PLAYWRIGHT_SHARD environment variable to enable blob reporting mode
Extracted reporter configuration logic into a reusable getPlaywrightReporters() function in playwright.config.ts
Implemented proper artifact management with appropriate retention periods
Added comprehensive caching strategy to reduce redundant setup work

Benefits

Reduces CI execution time by running tests in parallel across multiple shards
Minimizes redundant dependency installation across jobs through shared caching
Maintains full test coverage while improving build efficiency
Preserves backward compatibility with existing test infrastructure

The workflow now runs a Setup job once to prepare cached dependencies, then executes 12 parallel test jobs (3 OS × 4 shards) that leverage the shared cache, followed by a merge job that combines all test reports into a unified result.

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Co-authored-by: hudson-newey <33742269+hudson-newey@users.noreply.github.com>

hudson-newey · 2025-08-29T06:49:17Z

.github/workflows/tests.yml

+          name: blob-report-${{ matrix.os }}-${{ matrix.shard }}
+          path: blob-report/


Revert blob-report name change.

Suggested change

name: blob-report-${{ matrix.os }}-${{ matrix.shard }}

path: blob-report/

name: test-report-${{ matrix.os }}-${{ matrix.shard }}

path: test-report/

Changed the artifact name back to test-report-${{ matrix.os }}-${{ matrix.shard }} and updated path to test-report/ as requested. Commit 5ecd98a

.github/workflows/tests.yml

hudson-newey · 2025-08-29T06:53:36Z

.github/workflows/tests.yml

+          path: all-blob-reports/
+          pattern: blob-report-*


Suggested change

path: all-blob-reports/

pattern: blob-report-*

path: all-test-reports/

pattern: test-report-*

Updated to use path: all-test-reports/ and pattern: test-report-* as suggested. Commit 5ecd98a

hudson-newey · 2025-08-29T06:53:46Z

.github/workflows/tests.yml

+          path: all-blob-reports/
+          pattern: blob-report-*
+
+      - name: Merge Blob Reports


Suggested change

- name: Merge Blob Reports

- name: Merge Test Reports

Renamed to "Merge Test Reports" for clarity. Commit 5ecd98a

hudson-newey · 2025-08-29T06:53:56Z

.github/workflows/tests.yml

+            - recursive: true
+            - args: [--frozen-lockfile]
+
+      - name: Download All Blob Reports


Suggested change

- name: Download All Blob Reports

- name: Download All Test Reports

Renamed to "Download All Test Reports" for clarity. Commit 5ecd98a

hudson-newey · 2025-08-29T06:54:25Z

.github/workflows/tests.yml

+          pattern: blob-report-*
+
+      - name: Merge Blob Reports
+        run: pnpm exec playwright merge-reports --reporter=html,github all-blob-reports/


Suggested change

run: pnpm exec playwright merge-reports --reporter=html,github all-blob-reports/

run: pnpm exec playwright merge-reports --reporter=html,github all-test-reports/

Updated run command to use all-test-reports/ directory. Commit 5ecd98a

hudson-newey · 2025-08-29T06:55:14Z

playwright.config.ts

+  reporter: process.env.PLAYWRIGHT_SHARD
+    ? [
+        ["blob"],
+        ["github"],
+      ]
+    : [
+        // create a HTML report of the test results
+        // this is the best way to debug why tests are failing locally
+        [
+          "html",
+          {
+            outputFolder: "test-report",
+            open: "never",
+          },
+        ],
+        // print the test results out to the console.
+        // this can be useful for seeing why a test has failed in CI
+        isCi ? ["github"] : ["list"],
+      ],


This is a TypeScript file.

Extract this out to a function

Extracted the reporter configuration logic into a getPlaywrightReporters() function to improve code organization in the TypeScript file. Commit 5ecd98a

hudson-newey · 2025-08-29T06:56:22Z

@copilot Address review feedback

…ction Co-authored-by: hudson-newey <33742269+hudson-newey@users.noreply.github.com>

Copilot · 2025-08-29T07:01:21Z

@copilot Address review feedback

All review feedback has been addressed in commit 5ecd98a. Changes include: reverting artifact naming, adding retention settings, improving comments, renaming workflow steps, updating paths/patterns, and extracting TypeScript logic into a function.

hudson-newey · 2025-08-29T07:04:52Z

Good starter. I might have to do some minor formatting changes, but copilot seems to have (almost) nailed it

hudson-newey · 2025-08-29T07:07:39Z

@copilot Instead of cloning and pnpm installing dependencies on each testing shard, can you use a common base workflow task to set up the environment and use that as the base for all subsequent tests

Co-authored-by: hudson-newey <33742269+hudson-newey@users.noreply.github.com>

Copilot · 2025-08-29T07:13:33Z

@copilot Instead of cloning and pnpm installing dependencies on each testing shard, can you use a common base workflow task to set up the environment and use that as the base for all subsequent tests

Added a Setup job that runs once to prepare the environment and shared dependency cache across all test matrix jobs. Enhanced caching for both pnpm dependencies and Playwright browsers per OS. This reduces redundant setup work while maintaining the cross-platform testing matrix. Commit b5c7e9b

Initial plan

7d1950a

Copilot AI assigned Copilot and hudson-newey Aug 29, 2025

Copilot started work on behalf of hudson-newey August 29, 2025 06:39 View session

Copilot AI and others added 2 commits August 29, 2025 06:46

Implement test sharding in CI with Playwright

1824ebc

Co-authored-by: hudson-newey <33742269+hudson-newey@users.noreply.github.com>

Add documentation for test sharding

6aea9e2

Co-authored-by: hudson-newey <33742269+hudson-newey@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Can you complete issue #457 (Shard tests in CI). Perform a minimal number of changes to complete the task, and no not expand scope past the original issue description. Use pnpm~~ Implement test sharding in CI to reduce build time Aug 29, 2025

Copilot AI requested a review from hudson-newey August 29, 2025 06:51

Copilot finished work on behalf of hudson-newey August 29, 2025 06:51

hudson-newey requested changes Aug 29, 2025

View reviewed changes

Copilot started work on behalf of hudson-newey August 29, 2025 06:56 View session

hudson-newey added the CI label Aug 29, 2025

Address PR review feedback: revert naming, add retention, extract fun…

5ecd98a

…ction Co-authored-by: hudson-newey <33742269+hudson-newey@users.noreply.github.com>

Copilot AI requested a review from hudson-newey August 29, 2025 07:02

Copilot finished work on behalf of hudson-newey August 29, 2025 07:02

Copilot started work on behalf of hudson-newey August 29, 2025 07:07 View session

Optimize CI workflow with common setup job and enhanced caching

d7a0200

Co-authored-by: hudson-newey <33742269+hudson-newey@users.noreply.github.com>

Copilot AI changed the title ~~Implement test sharding in CI to reduce build time~~ Implement test sharding in CI with workflow optimization to reduce build time Aug 29, 2025

Copilot finished work on behalf of hudson-newey August 29, 2025 07:14

		name: blob-report-${{ matrix.os }}-${{ matrix.shard }}
		path: blob-report/

	- name: Download All Blob Reports
	- name: Download All Test Reports

	run: pnpm exec playwright merge-reports --reporter=html,github all-blob-reports/
	run: pnpm exec playwright merge-reports --reporter=html,github all-test-reports/

Implement test sharding in CI with workflow optimization to reduce build time #481

Are you sure you want to change the base?

Implement test sharding in CI with workflow optimization to reduce build time #481

Conversation

Copilot AI commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Features

Technical Implementation

Benefits

Uh oh!

hudson-newey Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hudson-newey Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

hudson-newey Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

hudson-newey Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

hudson-newey Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

hudson-newey Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

hudson-newey commented Aug 29, 2025

Uh oh!

Copilot AI commented Aug 29, 2025

Uh oh!

hudson-newey commented Aug 29, 2025

Uh oh!

hudson-newey commented Aug 29, 2025

Uh oh!

Copilot AI commented Aug 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Aug 29, 2025 •

edited

Loading