ci: report E2E preview test status back to PR commits

[michellepace]

Summary

• Add statuses: write permission and status reporting step to test-e2e-vercel.yml
• Reorder project setup guide so workflows run before configuring branch protection
• Replace project-specific "devflow" with generic "my_proj" in docs

Why

The repository_dispatch event runs in the context of main, not the PR branch. Without explicit status reporting, GitHub never associates the workflow result with the PR — causing required checks to block indefinitely.

Test plan

• Verify Run E2E Tests on Preview appears in PR checks after Vercel deploys
• After merge, add Run E2E Tests on Preview to branch ruleset
• Create a test PR to confirm it works as a required check

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
nextjs-base	Ready	Preview	Comment	Nov 29, 2025 4:37pm

[coderabbitai]

Summary by CodeRabbit

• Documentation

  • Updated project setup guide with new E2E testing and Vercel configuration procedures
  • Restructured workflow triggers and branch protection requirements
  • Added CodeRabbit setup instructions

• Chores

  • Enhanced CI/CD workflow to report test execution status to pull requests

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

This pull request updates the E2E Vercel workflow to report test statuses back to pull requests and revises the project setup documentation. The workflow now includes write permission for statuses and a reporting step that communicates test results. Documentation is restructured with updated project naming and enhanced instructions for E2E testing and branch protection configuration.

Changes

Cohort / File(s)	Summary
Workflow status reporting .github/workflows/test-e2e-vercel.yml	Adds write permission for statuses; introduces "Report status to PR" step that runs unconditionally after tests, mapping cancelled runs to error status, using Vercel deployment SHA as commit SHA, setting context to "Run E2E Tests on Preview", and linking to run URL.
Project setup documentation xdocs/project-setup.md	Replaces project name references from "devflow" to "my_proj"; restructures guide with new "Part 4: Trigger Initial Workflows" and consolidates CodeRabbit instructions into Part 6; introduces E2E on Vercel previews and CLI linking sequences; updates Branch Protection Ruleset with new status checks and "Do not require status checks on creation" option; removes "Verify Setup" and "Appendix" sections.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~18 minutes

• Workflow logic verification: Ensure status reporting correctly maps cancelled runs to error and uses the appropriate commit SHA from event payload
• Documentation coherence: Confirm restructured workflow instructions and section numbering are internally consistent across all parts
• Branch protection requirements: Verify new status checks align with updated workflow step naming and prerequisites

Possibly related PRs

• ci: add Vercel deployment with E2E tests on Preview #2: Modifies the Vercel E2E workflow with Playwright integration for preview testing
• docs: add setup guide and refresh README #8: Updates setup documentation with overlapping Vercel and E2E configuration instructions
• docs: project setup guide and README updates #9: Contains related branch protection and E2E setup configuration changes for project documentation

Poem

🐰 A workflow that reports with pride,
Status checks flowing far and wide,
Vercel previews now in view,
Documentation fresh and new,
From "devflow" to "my_proj" we stride!

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'ci: report E2E preview test status back to PR commits' directly and specifically captures the main change: adding status reporting for E2E preview tests to PR commits.
Description check	✅ Passed	The description clearly relates to the changeset, explaining the permissions addition, status reporting step, documentation reordering, and project name replacement with context for why these changes are needed.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/e2e-vercel-pr-status-and-docs

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 37c93c4 and 3cc4e1a.

📒 Files selected for processing (2)

• .github/workflows/test-e2e-vercel.yml (2 hunks)
• xdocs/project-setup.md (4 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Run E2E Tests

🔇 Additional comments (6)

.github/workflows/test-e2e-vercel.yml (2)

18-20: Permission addition is correct and necessary.

The statuses: write permission is required for the subsequent GitHub REST API call to create commit statuses. This is the minimal required permission.

---

87-103: The review comment is based on unfounded concerns—verification confirms the code is correct.

Both actions/github-script@v8 (latest maintained version 8.0.0 as of November 29, 2025) and the GitHub REST API repos.createCommitStatus() call are compatible and correctly implemented:

• v8 stability: Confirmed as the current maintained major version with active updates.
• API parameters: All required parameters (owner, repo, sha, state) and optional parameters (context, description, target_url) are correctly passed according to GitHub REST API documentation.
• State mapping: The cancelled → 'error' mapping is valid per API specification (state must be one of: error, failure, pending, success).

The original review comment correctly identified that the status reporting logic works as intended, so no changes are needed.

xdocs/project-setup.md (4)

7-24: Generic project name refactor is appropriate.

Updating "devflow" to "my_proj" throughout aligns with the template's goal of being easily adaptable to any project. The changes are consistent across clone and Vercel import steps.

---

28-54: E2E Vercel setup instructions are clear and comprehensive.

The new subsections 2.2 and 2.3 correctly guide users through:

• Creating a Vercel bypass secret for automation (prerequisite for Playwright access to protected deployments)
• Adding the secret to GitHub Actions (aligns with VERCEL_AUTOMATION_BYPASS_SECRET in the workflow)
• Optional Vercel CLI setup for local development

These instructions complement the workflow changes and address the environment variables used in the test step.

---

74-88: Restructuring to "Part 4: Trigger Initial Workflows" achieves the PR objective.

Moving the workflow trigger step before branch protection configuration is critical: it ensures all check names (including Run E2E Tests on Preview) are registered in GitHub before attempting to add them to the ruleset. The step-by-step instructions are clear (create test PR, wait for checks, verify deployments, merge).

---

112-118: Verify the branch protection setting label and completeness of required checks.

The checkbox Do not require status checks on creation and the inclusion of Run E2E Tests on Preview in the required checks list align with the workflow changes. However, I need to confirm:

1. That Do not require status checks on creation is the correct current GitHub UI label (UI terminology may vary across GitHub versions)
2. That the list of required checks is complete and matches all workflows that should gate merges to main

Can you verify that the GitHub branch protection rule "Do not require status checks on creation" is the correct label in your GitHub UI (version/region), and that it is the intended setting for your use case?

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}