feat(tests): enhance CI workflow and Playwright configuration#279
feat(tests): enhance CI workflow and Playwright configuration#279
Conversation
- Added a build step for the www app in the CI workflow to ensure the application is built before running tests. - Updated Playwright configuration to support different server modes for CI and local development, improving stability and performance. - Enhanced README documentation to clarify the web server configuration for testing environments. This update optimizes the testing process by ensuring a production build is used in CI, while allowing for a development server in local environments, thus improving developer experience and CI reliability.
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
WalkthroughAdds a pnpm build step to CI e2e tests and refines Playwright configuration and README to distinguish CI vs local webServer behavior, timeouts, reporters, and artifact capture policies. Changes
Sequence Diagram(s)sequenceDiagram
participant GH as GitHub Actions
participant Build as Build Step
participant Server as WebServer
participant Playwright as Playwright e2e
participant Artifacts as Videos/Screenshots
rect rgb(220,240,255)
Note over GH: CI / PR run
GH->>Build: install deps\n`pnpm --filter=www run build`
Build->>Server: start production server\n(long timeout, no reuse)
Server->>Playwright: serve app
Playwright->>Artifacts: capture only on failures
Playwright->>GH: report results
end
rect rgb(240,255,240)
Note over GH: Local developer run
GH->>Server: start dev server\n(reuse, hot-reload, short timeout)
Server->>Playwright: serve app
Playwright->>Artifacts: capture only on failures
Playwright->>GH: report results
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes
Possibly related PRs
Suggested labels
Poem
Pre-merge checks and finishing touches✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
bot
added
docs
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tooling/playwright-www/playwright.config.ts (1)
17-22: Validate the worker override before returning it.Returning
Number.parseInt(envWorkers, 10)directly means a malformed override (e.g.,"auto", empty string,"0") propagates asNaN/0, and Playwright refuses to start. Please guard against non-finite or non-positive values so we fall back to the default instead of crashing.Apply this diff to add the guard:
// Allow override via environment variable const envWorkers = process.env.PLAYWRIGHT_WORKERS; if (envWorkers) { - return Number.parseInt(envWorkers, 10); + const overridden = Number.parseInt(envWorkers, 10); + if (Number.isFinite(overridden) && overridden > 0) { + return overridden; + } }tooling/playwright-www/README.md (1)
62-66: Keep the screenshot guidance in sync with the config.The config now sets
screenshot: "only-on-failure", so screenshots aren’t produced for every run. Please reword this bullet (and any similar spots) to note they’re captured only on failure to avoid surprising folks looking for happy-path screenshots.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
.github/workflows/tests.yml(1 hunks)tooling/playwright-www/README.md(1 hunks)tooling/playwright-www/playwright.config.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/README.md
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Document environment requirements in README files
Files:
tooling/playwright-www/README.md
🧠 Learnings (1)
📚 Learning: 2025-09-12T06:23:45.463Z
Learnt from: CR
Repo: yamcodes/arkenv PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-12T06:23:45.463Z
Learning: Applies to **/*.test.ts : In tests, save and restore process.env in setup/teardown (beforeEach/afterEach) to avoid leakage between cases
Applied to files:
tooling/playwright-www/playwright.config.ts
🧬 Code graph analysis (2)
.github/workflows/tests.yml (5)
tooling/playwright-www/tests/integrations.test.ts (1)
test(4-157)tooling/playwright-www/tests/morphs.test.ts (2)
test(4-149)page(106-126)tooling/playwright-www/tests/docs-navigation.test.ts (1)
test(4-181)tooling/playwright-www/tests/examples.test.ts (1)
test(4-94)tooling/playwright-www/tests/homepage.test.ts (1)
test(4-205)
tooling/playwright-www/playwright.config.ts (3)
tooling/playwright-www/tests/integrations.test.ts (3)
test(4-157)page(82-97)page(18-41)tooling/playwright-www/tests/quickstart.test.ts (4)
page(78-86)test(4-181)page(158-165)page(18-34)tooling/playwright-www/tests/homepage.test.ts (1)
test(4-205)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: test-e2e (e2e)
- GitHub Check: test-e2e (a11y)
- Changed the CI workflow to use `pnpm run test` instead of `pnpm turbo run test` for consistency. - Updated the build command in the CI workflow to `pnpm run build --filter=www...` to ensure all dependencies are built. - Revised Playwright configuration comments to reflect the new build command requirements. - Enhanced README documentation to clarify the build requirements for running tests, emphasizing the need to build the `arkenv` package. These changes improve the clarity and reliability of the testing process in both CI and local environments.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tooling/playwright-www/playwright.config.ts (1)
43-43: Consider extracting the port number to a constant.The port "3000" is hardcoded in three places (baseURL on line 43, and env.PORT on lines 78 and 88). While this works correctly, extracting it to a constant would improve maintainability.
Apply this diff to extract the port constant:
+const PORT = "3000"; + export default defineConfig({ testDir: "./tests", fullyParallel: true, forbidOnly: isCi, retries: isCi ? 2 : 0, workers: getWorkers(), // Reporter: dot/html for CI (less noise), list/html for local (immediate feedback) reporter: isCi ? [["dot"], ["html", { open: "never" }]] : [["list"], ["html", { open: "on-failure" }]], // Test timeout: 60s for CI (networkidle waits, axe-core scans), 30s for local timeout: isCi ? 60_000 : 30_000, // Explicit expect timeout prevents flaky assertions expect: { timeout: 5_000 }, use: { - baseURL: "http://localhost:3000", + baseURL: `http://localhost:${PORT}`, trace: "on-first-retry", // Capture videos and screenshots only on failure to reduce CI artifact size video: "retain-on-failure", screenshot: "only-on-failure", }, projects: [ { name: "chromium", use: { ...devices["Desktop Chrome"] }, }, { name: "firefox", use: { ...devices["Desktop Firefox"] }, }, { name: "webkit", use: { ...devices["Desktop Safari"] }, }, ], // Web server configuration // CI: Use production server (next start) for stability - requires pre-build in workflow // Local: Use dev server (next dev) for hot reload and better DX webServer: isCi ? { // CI: Production server (faster, more stable, matches production) // Requires: pnpm run build --filter=www... in GitHub Actions before tests command: "pnpm --filter=www run start", - url: "http://localhost:3000", + url: `http://localhost:${PORT}`, reuseExistingServer: false, timeout: 120_000, // 2min should be enough for production server stdout: "pipe", stderr: "pipe", - env: { PORT: "3000" }, + env: { PORT }, } : { // Local: Dev server (hot reload, sourcemaps, better DX) command: "pnpm --filter=www run dev", - url: "http://localhost:3000", + url: `http://localhost:${PORT}`, reuseExistingServer: true, timeout: 120_000, // 2min for dev server cold start stdout: "pipe", stderr: "pipe", - env: { PORT: "3000" }, + env: { PORT }, }, });Also applies to: 78-78, 88-88
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
.github/workflows/tests.yml(4 hunks)tooling/playwright-www/README.md(1 hunks)tooling/playwright-www/playwright.config.ts(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/tests.yml
🧰 Additional context used
📓 Path-based instructions (1)
**/README.md
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Document environment requirements in README files
Files:
tooling/playwright-www/README.md
🧠 Learnings (4)
📓 Common learnings
Learnt from: CR
Repo: yamcodes/arkenv PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-12T06:23:45.463Z
Learning: When contributing, ensure tests pass (pnpm test), lint passes (pnpm check/fix), and create a changeset (pnpm changeset) before merging
📚 Learning: 2025-09-12T06:23:45.463Z
Learnt from: CR
Repo: yamcodes/arkenv PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-12T06:23:45.463Z
Learning: Applies to **/*.test.ts : Test both success and failure cases for environment validation and utilities
Applied to files:
tooling/playwright-www/README.md
📚 Learning: 2025-09-12T06:23:45.463Z
Learnt from: CR
Repo: yamcodes/arkenv PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-12T06:23:45.463Z
Learning: Applies to **/*.test.ts : Verify both runtime behavior and TypeScript types in tests
Applied to files:
tooling/playwright-www/README.md
📚 Learning: 2025-09-12T06:23:45.463Z
Learnt from: CR
Repo: yamcodes/arkenv PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-12T06:23:45.463Z
Learning: Applies to **/*.test.ts : In tests, save and restore process.env in setup/teardown (beforeEach/afterEach) to avoid leakage between cases
Applied to files:
tooling/playwright-www/playwright.config.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: test-e2e (e2e)
- GitHub Check: test-e2e (a11y)
🔇 Additional comments (8)
tooling/playwright-www/playwright.config.ts (4)
34-37: LGTM: Appropriate reporter configuration for each environment.The conditional reporter setup provides concise output for CI (dot) and detailed feedback for local development (list), with appropriate HTML artifact handling.
38-41: LGTM: Well-reasoned timeout configuration.The timeout values are appropriate for each environment. The explicit expect timeout of 5s is a good defensive measure against flaky assertions.
44-47: LGTM: Efficient artifact capture strategy.Capturing videos and screenshots only on failure is an effective approach to reduce CI artifact size while maintaining debuggability.
65-89: Configuration verified — no changes needed.The GitHub Actions workflow correctly includes the build step (
pnpm run build --filter=www...) before running Playwright tests, confirming that the production server configuration inplaywright.config.tsis properly supported by the CI pipeline.tooling/playwright-www/README.md (4)
68-79: LGTM: Clear documentation of build requirements.The Build Requirements section clearly explains why the arkenv package must be built before tests and provides the necessary commands for both CI and local environments. This aligns with the coding guideline to document environment requirements in README files.
Based on coding guidelines.
80-101: LGTM: Comprehensive server configuration documentation.The Web Server Configuration section clearly documents the different server modes for CI and local environments, with well-reasoned rationale for using production builds in CI. The documentation aligns perfectly with the implementation in playwright.config.ts.
102-126: LGTM: Thorough troubleshooting guide.The Troubleshooting section provides practical, actionable solutions for the most common issues, with environment-specific guidance for CI vs local development. The solutions are clear and include specific commands.
127-134: LGTM: Accurate documentation of reporter and artifacts.The Reporter and Artifacts section accurately documents the configuration implemented in playwright.config.ts, clearly explaining the different behaviors for CI and local environments.
1 participant
This update optimizes the testing process by ensuring a production build is used in CI, while allowing for a development server in local environments, thus improving developer experience and CI reliability.
Summary by CodeRabbit
Tests
Documentation