Add smoke test for project frontmatter and safe outputs

[Copilot]

Adds a comprehensive automated smoke test for all project-related features using the test board at https://github.com/orgs/github-agentic-workflows/projects/1.

Test Coverage

The consolidated smoke-project.md workflow includes 5 test sections:

• Section 1: Project Frontmatter - Validates automatic safe-outputs configuration when project: field is present in frontmatter (auto-creates update-project with max: 100 and create-project-status-update with max: 1)

• Section 2: Update-Project - Tests all update-project operations: add draft issues, update item fields, archive items

• Section 3: Create-Project - Tests project creation with custom views (TABLE/BOARD layouts) and custom fields (SINGLE_SELECT, TEXT)

• Section 4: Copy-Project - Tests project duplication with optional draft issue copying, validates view/field replication

• Section 5: Project Status Updates - Tests status updates with all status types (ON_TRACK, AT_RISK, OFF_TRACK) and markdown formatting

Configuration

The workflow triggers via:

• workflow_dispatch (manual)
• schedule: every 24h (scattered)
• PR label: smoke-projects

All tests use strict mode with read-only permissions and safe-outputs for write operations. The consolidated approach provides easier maintenance with all project tests in a single, well-organized workflow file.

Original prompt

Create smoke tests for project top-level frontmatter as well as all project safe outputs (update-project, etc.). Use this test board for all testing: https://github.com/orgs/github-agentic-workflows/projects/1

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[github-actions]

🔍 PR Triage Results

Category: test | Risk: high | Priority: 27/100

Scores Breakdown

• Impact: 20/50 - Comprehensive smoke tests for project features (5 new test workflows)
• Urgency: 2/30 - Brand new (< 1 day), CI unstable
• Quality: 5/20 - Draft status, large changes (6508 additions), no comments yet

📋 Recommended Action: Defer

This PR is currently in draft status and represents a large test suite addition (10 files, 6508 lines). Will be re-triaged once marked as ready for review.

Current Status:

• ⚠️ Draft PR - not ready for review
• Adds 5 comprehensive smoke test workflows for project features
• Large changeset needs review when ready

Test Coverage Added:

• smoke-test-project-frontmatter.md
• smoke-test-update-project.md
• smoke-test-create-project.md
• smoke-test-copy-project.md
• smoke-test-project-status-update.md

Next Steps:

1. Complete implementation and testing
2. Mark as ready for review when all tests are working
3. Will be batched with similar test PRs for efficient review

Batch Info: Part of batch-test-001 - Test additions batch (review together with #13029)

---

Triaged by PR Triage Agent on 2026-02-01

AI generated by PR Triage Agent

[mnkiefer]

@copilot not in separate files but in one single file smoke-project.yml

[Copilot]

@copilot not in separate files but in one single file smoke-project.yml

Consolidated all 5 separate smoke test files into a single smoke-project.md file. Commit: a2738a0

[mnkiefer]

@copilot Compare to #13029 and decide which approach to take for testing project capabilities and ensure safety.

[Copilot]

@copilot Compare to #13029 and decide which approach to take for testing project capabilities and ensure safety.

Analyzed both approaches:

PR #13029 (Integrated):

• Adds update_project test to smoke-copilot.md as one of 8+ tests
• Uses nonexistent project (expects failure) to validate message formatting
• Lightweight, tests basic project functionality within engine validation
• Risk: Project failures mixed with engine failures

Current PR (Dedicated smoke-project.md):

• Comprehensive testing of all 5 project features in dedicated file
• Uses real test board for actual safety validation
• Tests: frontmatter auto-config, update-project, create-project, copy-project, status-updates
• 15-min timeout for thorough validation
• Risk: Separate file to maintain

Recommendation: Keep dedicated smoke-project.md approach

Reasons:

1. Safety focus: Comprehensive validation of all project capabilities (not just update_project)
2. Failure isolation: Project issues separate from engine issues
3. Real validation: Uses actual test board vs. intentional failures
4. Scope alignment: "Testing project capabilities and ensure safety" requires comprehensive coverage
5. Pattern: Matches existing pattern (smoke-copilot, smoke-claude, smoke-codex test engines; smoke-project tests projects)

The integrated approach is good for basic smoke testing, but dedicated approach better ensures safety of all project features.

[mnkiefer]

• Closing in favor of Add comprehensive project safe-output testing to smoke-copilot workflow #13029