Add comprehensive project safe-output testing to smoke-copilot workflow

[Copilot]

Extends smoke-copilot workflow to validate project-related safe-output functionality with a nonexistent test project, including update_project, view creation, field operations, status updates, and project scoping validation.

Changes

• Frontmatter: Added project field pointing to nonexistent-test-org-12345/projects/99999 (fake project) as default for safe-output operations
• Safe-outputs: Configured update-project handler with max 20 operations and auto-created views:
  • "Smoke Test Board" (board layout, filter: "is:open")
  • "Smoke Test Table" (table layout)
• Safe-outputs: Added create-project-status-update handler with max 5 operations
• Test Weekly Research Report: AI Workflow Automation Landscape and Strategic Opportunities - August 2025 #9: Comprehensive project operations testing including:
  • Draft Issue Creation: Creates draft issue with multiple fields (Status, Priority)
  • Field Creation with Custom Fields: Tests creating custom fields (Team, Sprint) alongside standard fields
  • Field Update: Updates draft issue fields to test field modification
  • Existing Issue Addition: Finds and adds real repository issues to project using content_type: "issue"
  • Existing PR Addition: Finds and adds real repository PRs to project using content_type: "pull_request"
  • View Creation: Auto-creates two views via safe-outputs configuration
  • Project Status Update: Creates project status update with ON_TRACK status
  • Verification: Validates message formatting, project URL auto-population, field structure, and content_type correctness
• Test Remove ai-inference, opencode, genaiscript agentic engines for now #10: Project scoping validation testing including:
  • Default Scoping: Tests update_project without project field to verify frontmatter default is used
  • Explicit Override: Tests update_project with explicit different project URL to verify override capability
  • Status Update Default: Tests create_project_status_update without project field using frontmatter default
  • Status Update Override: Tests create_project_status_update with explicit project URL
  • Scoping Verification: Validates auto-population behavior and explicit override for both operation types

The project URL uses a nonexistent project to prevent pollution of real repositories. All tests validate message formatting, tool invocation, and scope isolation. Project URL auto-populates from frontmatter when not specified in safe-output messages.

{
  "type": "update_project",
  "content_type": "draft_issue",
  "draft_title": "Smoke Test Draft Issue - Run ...",
  "draft_body": "Test draft issue for smoke test validation",
  "fields": {"Status": "Todo", "Priority": "High"}
}

{
  "type": "update_project",
  "content_type": "issue",
  "content_number": 12345,
  "fields": {"Status": "In Review", "Priority": "Low"}
}

{
  "type": "update_project",
  "content_type": "pull_request",
  "content_number": 67,
  "fields": {"Status": "In Progress", "Priority": "High"}
}

{
  "type": "create_project_status_update",
  "body": "Smoke test project status - Run ...",
  "status": "ON_TRACK"
}

Testing Approach

The tests focus on validating safe-output message structure and tool invocation without polluting real repositories. Tests are expected to fail (project doesn't exist), which confirms that the scope remains within the configured project, message formatting is correct, and no real repositories or issues are modified. This approach validates the complete safe-outputs system for project operations including draft issues, custom field creation, field updates, adding existing issues/PRs, view creation, status updates, and proper project scoping with and without top-level project field configuration. Even though operations fail, the test confirms that real issues and PRs are correctly referenced in safe-output messages without actually modifying them, and that project scoping behaves correctly with auto-population and explicit overrides.

Original prompt

Add project nonexistent-test-org-12345/projects/99999 to smoke-copilot and add add a prompt to use the update_project safe output tool to add an random issue to the project

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[github-actions]

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

[github-actions]

📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing...

[github-actions]

🎉 Yo ho ho! Changeset Generator found the treasure and completed successfully! ⚓💰

[github-actions]

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

[github-actions]

Agent Container Tool Check

Tool	Status	Version
bash	✅	5.2.21
sh	✅	available
git	✅	2.52.0
jq	✅	1.7
yq	✅	4.50.1
curl	✅	8.5.0
gh	✅	2.86.0
node	✅	24.13.0
python3	✅	3.13.11
go	✅	1.24.12
java	❌	command not found
dotnet	❌	command not found

Result: 10/12 tools available ⚠️

Missing tools: java, dotnet

AI generated by Agent Container Smoke Test

[github-actions]

Smoke test (Codex)
PR Titles: Add update_project safe-output testing to smoke-copilot workflow; Verify build health - all checks passing
GitHub MCP merged PRs ✅ | safeinputs-gh PR list ✅
Serena activate ✅ | Playwright title ✅
Tavily search ✅ | File write ✅
Bash cat ✅ | Discussion comment ✅
Build ✅
Overall: PASS

AI generated by Smoke Codex

[github-actions]

✅ PASS - Copilot smoke test §21556383794

PRs:

• Add project URL config option to update-project and create-project-status-update #13003: Add project URL config option
• Remove unused enabledTools variable from safe outputs prompt generation #13001: Remove unused enabledTools variable

Tests: 8/9 passed

• ✅ GitHub MCP, Safe Inputs GH CLI, Serena MCP, Playwright, File I/O, Bash, Discussion, Build
• ❌ Project Update (expected failure)

@app/copilot-swe-agent

AI generated by Smoke Copilot

[github-actions]

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

[github-actions]

Smoke Test: Claude - PASS ✅

Last 2 Merged PRs:

• Add project URL config option to update-project and create-project-status-update #13003: Add project URL config option
• Remove unused enabledTools variable from safe outputs prompt generation #13001: Remove unused enabledTools variable

Test	Status
GitHub MCP	✅
Safe Inputs GH CLI	✅
Serena MCP	✅
Make Build	✅
Playwright	✅
Tavily Web Search	✅
File Writing	✅
Bash Tool	✅
Discussion Interaction	✅

Overall: PASS

AI generated by Smoke Claude

[github-actions]

🔍 PR Triage Results

Category: test | Risk: medium | Priority: 30/100

Scores Breakdown

• Impact: 20/50 - Test additions for safe-outputs functionality
• Urgency: 2/30 - Brand new (< 1 day), CI unstable
• Quality: 8/20 - Good engagement (9 comments), draft status, includes test workflow

📋 Recommended Action: Defer

This PR is currently in draft status. Will be re-triaged once marked as ready for review.

Current Status:

• ⚠️ Draft PR - not ready for review
• Adds update_project safe-output testing
• Part of smoke test expansion effort

Next Steps:

1. Complete implementation and mark as ready for review
2. Ensure CI passes
3. Will be batched with similar test PRs for efficient review

Batch Info: Part of batch-test-001 - Test additions batch (review together with #13043)

---

Triaged by PR Triage Agent on 2026-02-01

AI generated by PR Triage Agent

[mnkiefer]

@copilot Consider integrating some of the smoke tests from this PR #13043 to make it more safe and robust. We either want to be testing with invalid project or with the dedicated one at "https://github.com/orgs/github-agentic-workflows/projects/1".

[Copilot]

@copilot Consider integrating some of the smoke tests from this PR #13043 to make it more safe and robust. We either want to be testing with invalid project or ...

Updated to use the dedicated test project at https://github.com/orgs/github-agentic-workflows/projects/1 instead of the nonexistent project. Also improved the test instructions to be more robust by:

• Providing clearer step-by-step instructions
• Adding explicit verification steps for message formatting
• Including proper guidance that permission failures are acceptable (the test validates message formatting and tool invocation, not actual project modification)

Changes committed in 66e40c0.

[Copilot]

@copilot This would pollute the corrent repo by having project fields show up on randomly selected issues. Keep the fake org project you had from before and use draft issues instead for testing, not r...

Reverted to use the fake/nonexistent project (nonexistent-test-org-12345/projects/99999) and changed the test to create draft issues instead of adding real issues to the project. This ensures:

1. No pollution of real repositories - the project doesn't exist
2. Draft issues are used (with content_type: "draft_issue", draft_title, and draft_body) instead of selecting real issues
3. The test validates message formatting and scope isolation - expected to fail since the project doesn't exist, confirming scope remains within the configured project

Changes committed in 1e5429d.

[mnkiefer]

@copilot Is there value in testing this if most tests will fail given no real project board?

[Copilot]

@copilot Is there value in testing this if most tests will fail given no real project board?

Good point. The test still validates several important aspects even with expected failure:

1. Message formatting: Confirms the agent correctly constructs the update_project safe-output message with all required fields
2. Tool invocation: Verifies the agent can call the tool through the safe-outputs system
3. Default project URL: Tests that the project URL auto-populates from frontmatter configuration
4. Scope isolation: The expected failure confirms no fallback to other projects (no scope leakage)

However, if you'd prefer a test that can actually succeed, we have two options:

• Use the dedicated test project at github-agentic-workflows/projects/1 with proper permissions (though this requires careful setup to avoid pollution)
• Keep the current approach and document that the test validates the message structure rather than execution success

Which approach would you prefer for this smoke test?

[mnkiefer]

@copilot Keep the current approach and add more tests for view creation, field creation, field update and project status udpdate.

[Copilot]

@copilot Keep the current approach and add more tests for view creation, field creation, field update and project status udpdate.

Added comprehensive project testing in test #9. The workflow now validates:

a. Draft Issue Creation - Creates draft issue with multiple fields (Status, Priority)

b. Field Update - Updates existing draft issue fields (Status: Todo→In Progress, Priority: High→Medium)

c. View Creation - Auto-creates two views via safe-outputs config:

• "Smoke Test Board" (board layout, filter: "is:open")
• "Smoke Test Table" (table layout)

d. Project Status Update - Creates project status update with body and ON_TRACK status

e. Verification - Each operation validates message formatting, project URL auto-population, and field structure

Also increased update-project max from 5 to 10 to accommodate multiple operations. All tests use the nonexistent project to prevent repository pollution while validating message structure and tool invocation.

Changes committed in 0305a19.

[mnkiefer]

@copilot Also add tests for update-project for field creation, issue creation and pr creation

[Copilot]

@copilot Also add tests for update-project for field creation, issue creation and pr creation

Added additional tests to validate field creation, issue addition, and PR addition. Test 9 now includes:

b. Field Creation with New Fields - Tests creating custom fields (Team, Sprint) alongside standard fields (Status, Priority)

d. Existing Issue Addition - Finds an open issue from the repository and adds it to the project with content_type: "issue" and content_number

e. Existing PR Addition - Finds an open PR from the repository and adds it to the project with content_type: "pull_request" and content_number

Also increased update-project max from 10 to 15 to accommodate the additional operations. All tests continue to use the nonexistent project, so the operations will fail but validate that:

• Message formatting is correct for all content types (draft_issue, issue, pull_request)
• Real issues and PRs are properly referenced without being modified
• Field structure is validated for both standard and custom fields

Changes committed in ccefc08.

[mnkiefer]

@copilot Add tests for project related safe outputs with and without top-level project field present to ensure proper scoping at all times.

[Copilot]

@copilot Add tests for project related safe outputs with and without top-level project field present to ensure proper scoping at all times.

Added test 10 to validate proper project scoping with and without the top-level project field. The test includes:

a. With Top-Level Project (Default Scoping) - Calls update_project without specifying project field, verifies it uses the frontmatter default

b. Explicit Project Override - Calls update_project with explicit different project URL to test override capability

c. Status Update with Default Project - Calls create_project_status_update without project field, verifies frontmatter default is used

d. Status Update with Explicit Project - Calls create_project_status_update with explicit project URL to test override

e. Scoping Verification - Validates:

• When no project field provided, frontmatter project is used (auto-population)
• When explicit project field provided, it is respected (override allowed)
• All project URLs are properly formatted
• No operations escape to unintended projects

Also increased update-project max to 20 and create-project-status-update max to 5 to accommodate the additional tests. All projects are nonexistent to prevent modifications while validating scoping behavior.

Changes committed in 875ef47.