Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jan 29, 2026

The ci-coach workflow lacked explicit guardrails against anti-patterns (swallowing test errors, ignoring failures) and wasn't leveraging the canary job's test coverage analysis.

Changes

Phase 2: Test Coverage Analysis

  • Canary job integration: Instructs workflow to review test-coverage-analysis artifact comparing all-tests.txt vs executed-tests.txt to identify missing tests in CI matrix
  • Test suite integrity validation: Verify test failures propagate correctly (non-zero exit codes, no error suppression in artifacts)

Important Guidelines: Test Code Integrity

Added critical section prohibiting:

  • Error suppression patterns (|| true, set +e, || echo "ignoring")
  • Modifying test files to hide failures
  • Commenting out assertions or skipping tests without justification

Explicit DO/DON'T guidance: fix root cause → ✅ | suppress errors → ❌

### Test Code Integrity (CRITICAL)

**NEVER MODIFY TEST CODE TO HIDE ERRORS**

The CI Coach workflow must NEVER alter test code (`*_test.go` files) in ways that:
- Swallow errors or suppress failures
- Add error suppression patterns like `|| true`, `|| :`, or `|| echo "ignoring"`
- Wrap test execution with `set +e` or similar error-ignoring constructs

Safety Checks

Reinforced test integrity requirements in validation checklist.

Original prompt

Update ci-coach agentic workflow:

  • it should NEVER EVER change the test code to hide errors by swallowing errors
  • it should determine if the test suite is actually testing anything. For example, if tests are reported as failing, the test suite should fail as well.
  • it should make sure all tests are executed and use the canary job outputs to determine which ones are missing

Custom agent used: agentic-workflows
GitHub Agentic Workflows (gh-aw) - Create, debug, and upgrade AI-powered workflows with intelligent prompt routing


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits January 29, 2026 00:40
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
- Add canary job output instructions to detect missing tests
- Add test suite integrity validation requirements
- Add strict prohibition against modifying test code to hide errors
- Ensure CI Coach never swallows errors or suppresses test failures

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Update CI coach agentic workflow to improve error handling ci-coach: Add test integrity safeguards and canary job integration Jan 29, 2026
Copilot AI requested a review from pelikhan January 29, 2026 00:50
@pelikhan pelikhan marked this pull request as ready for review January 29, 2026 00:51
@pelikhan pelikhan merged commit 83a58c2 into main Jan 29, 2026
@pelikhan pelikhan deleted the copilot/update-ci-coach-workflow branch January 29, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants