feat: implement issue #1364 — [Fleet Monitor] petry-projects/.github-private — .github/workflows/test.yml

[don-petry]

User description

Closes #1364

Implemented by dev-lead agent. Please review.

---

CodeAnt-AI Description

Make the test workflow resilient to temporary install failures

What Changed

• The test workflow now retries package and tool installs, so brief PyPI or apt mirror issues do not fail the run.
• Python dependency installation is pinned and limited to binary packages, which avoids falling back to a flaky source build.
• A regression test now checks that these safeguards stay in place.

Impact

✅ Fewer flaky CI failures
✅ More reliable test runs
✅ Fewer install-related workflow breaks

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

[codeant-ai]

🤖 CodeAnt AI — Review Status

Status	Commit	Started (UTC)	Finished (UTC)
✅ Incremental review completed	e9f064f	Jul 24, 2026 · 14:08	14:08
✅ Reviewed your PR	f68f1e8	Jul 23, 2026 · 09:21	09:24

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[coderabbitai]

Warning

Review limit reached

@don-petry, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 54 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ee155ed4-7585-4ba4-83c3-346e227f068a

📥 Commits

Reviewing files that changed from the base of the PR and between d692255 and e9f064f.

📒 Files selected for processing (2)

• .github/workflows/test.yml
• tests/test_test_workflow_resilience.sh

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-lead/issue-1364-20260723-0915

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qodo-code-review]

PR Summary by Qodo

Harden CI dependency installs with retries and add regression guard (issue #1364)

✨ Enhancement 🧪 Tests ⚙️ Configuration changes 🕐 10-20 Minutes

AI Description

• Add bounded retries around pip/apt installs to reduce CI flakes (issue #1364).
• Pin PyYAML and force binary-only installs to avoid flaky source builds.
• Add a workflow guard test to prevent resilience regressions.

Diagram

graph TD
  WF[".github/workflows/test.yml"] --> Install(["Install deps step"]) --> Retry(["retry() wrapper"]) --> Fetches(["pip + apt fetches"])
  Retry --> Pin(["pyyaml pinned + binary-only"])
  Guard["tests/test_test_workflow_resilience.sh"] --> Parse(["yq parse workflow"]) --> Assert(["assert retry/pin rules"]) --> WF

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Use built-in retry flags (pip/apt) instead of a custom retry()

• ➕ Less custom shell logic embedded in the workflow
• ➕ Leverages tool-native retry behavior (e.g., pip --retries, apt Acquire::Retries)
• ➖ Retry mechanisms differ across tools; harder to keep behavior consistent
• ➖ Some retries are less visible/auditable than an explicit wrapper

2. Prebake dependencies into a container or use action-based installs

• ➕ Removes most network variability from workflow runtime
• ➕ Potentially faster and more reproducible runs
• ➖ Higher maintenance overhead (image updates/action versioning)
• ➖ May be overkill for a small set of dependencies

Recommendation: The chosen approach (explicit bounded retry wrapper + pin/binary-only pip install) is a pragmatic fix for transient network flakes and keeps the behavior easy to audit. Native retry flags were considered, but the wrapper provides uniform handling across pip and apt while the new guard test prevents regressions.

Files changed (2) +98 / -2

Tests (1) +77 / -0

test_test_workflow_resilience.shNew bash test to enforce workflow install-step resilience rules +77/-0

New bash test to enforce workflow install-step resilience rules

• Adds a regression test that parses the workflow using yq, locates the "Install test dependencies" step, and asserts retry wrapping plus pinned/binary-only pip usage. Includes a guard for hyphenated job key parsing to avoid yq/jq expression pitfalls.

tests/test_test_workflow_resilience.sh

Other (1) +21 / -2

test.ymlAdd retry-hardened dependency installs and run a resilience guard +21/-2

Add retry-hardened dependency installs and run a resilience guard

• Replaces direct pip/apt installs with a bounded retry() helper to mitigate transient network failures. Pins PyYAML to a specific version and forces binary-only installs, then adds a new guard step to verify these invariants.

.github/workflows/test.yml

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-07-23T10:24:15Z.

[gemini-code-assist]

Code Review

This pull request introduces a regression guard script tests/test_test_workflow_resilience.sh to verify that the dependency installation step in the GitHub Actions workflow is resilient to transient network failures. The reviewer suggested several improvements to make the script more robust and portable: normalizing multi-line workflow commands to prevent line-by-line grep matching failures, replacing the non-portable \s regex shorthand with POSIX-compliant [[:space:]] for macOS compatibility, making the pyyaml check case-insensitive, and explicitly asserting that grep exits with status 1 when verifying the absence of a pattern to avoid false passes.

[don-petry]

Dev-Lead — fix-bot-comment (applied)

Changes committed and pushed.

[donpetry-bot]

CI checks on this PR are still running. Once they complete, re-mention @donpetry-bot to trigger a fresh review.

Posted by the donpetry-bot PR-review cascade.

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 48 rules

1. Binary-only PyYAML fragility ✗ Dismissed 🐞 Bug ☼ Reliability

Description

The Tests workflow installs an exact PyYAML version with --only-binary=:all: while running on
ubuntu-latest without pinning Python, so a future runner/Python update can break CI if
pyyaml==6.0.3 lacks a compatible wheel (pip will refuse an sdist fallback). This turns a
previously recoverable install into a deterministic failure mode unrelated to network flakiness.

Code

.github/workflows/test.yml[R41-43]

+          retry pip install "pyyaml==6.0.3" --only-binary=":all:" --quiet
+          retry sudo apt-get update
+          retry sudo apt-get install -y yq

Relevance

⭐⭐ Medium

Team accepts CI pinning but has rejected some PyYAML pin requests; binary-only wheel fragility is
plausible but intent was flake reduction.

PR-#284
PR-#276

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The job runs on ubuntu-latest and does not set up/pin a Python version, but it installs an exact
PyYAML version while forbidding source distributions, which can fail after runner image/Python
updates when a matching wheel is unavailable.

.github/workflows/test.yml[12-16]
.github/workflows/test.yml[24-43]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow uses `pip install "pyyaml==6.0.3" --only-binary=":all:"`, which requires a compatible wheel for that exact version. Because the job runs on `ubuntu-latest` and does not pin Python, future runner image updates can change the default Python version and cause this step to fail if no wheel is available.

## Issue Context
This PR is addressing CI flakiness in network installs by adding bounded retries and making the PyYAML install reproducible. The current flags improve determinism, but also couple CI to wheel availability for a single PyYAML release on whatever Python happens to ship with `ubuntu-latest`.

## Fix Focus Areas
- .github/workflows/test.yml[12-43]

## Recommended fix
Implement one of:
1) Pin Python explicitly (recommended): add `actions/setup-python` with a specific `python-version` known to have a `pyyaml==6.0.3` wheel, keeping `--only-binary=:all:`.
2) Allow fallback when needed: replace `--only-binary=:all:` with `--prefer-binary` (or drop `--only-binary`) so CI can still proceed if wheels are temporarily unavailable for a new Python ABI.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[don-petry]

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

[don-petry]

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning

- ✓ 0 New issues
- ✓ 0 Accepted issues  
- ✓ 0 Security Hotspots
- ✓ 0.0% Coverage/Duplication on New Code
**Tier 1 blocker check:**
All CI checks that have completed show `success` (except ecosystem-specific audits which are skipped). The one in-progress check is `bats` (testing). No checks show `failure`, `timed_out`, `cancelled`, `action_required`, or `stale`.
For reviews, none show `CHANGES_REQUESTED` state. The automated review bot (`donpetry-bot`) approved the PR with `DISMISSED` state, noting "All 8 review threads (CodeAnt, Gemini, Qodo, CodeRabbit) are resolved."
---
**Summary:**
No actionable issues to address. The SonarCloud Quality Gate passed with 0 new issues, and there are zero Tier 1 blockers (no CI failures or change requests). The PR is in good standing pending completion of the in-progress `bats` test suite.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-07-23T10:47:13Z.

[don-petry]

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning

Issues addressed: 0
- Quality gate passed; no specific issues to address
Files changed: none
Skipped (informational): Quality gate status report
```
The PR is ready to merge — all automated checks pass, all review threads are addressed or resolved, and there are no outstanding code quality or security issues identified by the bot.

[donpetry-bot]

Automated review — APPROVED ✓

Risk: MEDIUM
Reviewed commit: cd3b2a11dfb658c469527f0dde91f7b3708d69c6
Review mode: triage-approved (single reviewer)

Summary

Hardens the flaky "Install test dependencies" step in .github/workflows/test.yml (issue #1364, 14.8% failure rate) by wrapping the three network fetches (pip install, apt-get update, apt-get install) in a bounded 3-attempt retry helper, pinning pyyaml==6.0.3 with --only-binary to avoid flaky source builds, and adding tests/test_test_workflow_resilience.sh as a regression guard that asserts the retry wrapper, per-occurrence wrapping, the version pin, and binary-only install remain in place. Change is tightly scoped to the two files and matches the triage tier's low-risk assessment; MEDIUM only because it modifies workflow logic.

Linked issue analysis

Closes #1364 (Fleet Monitor: test.yml failing 14.8% of runs, threshold 10%). The only non-deterministic step in that workflow is the dependency install; this PR bounds retries on every network fetch and eliminates the pip source-build fallback, directly and substantively addressing the flake source. A regression guard test enforces the safeguards going forward.

Findings

No blocking findings.

• Review threads: 10 threads total; 8 marked resolved. Two graphite-app threads remain flagged unresolved on GitHub, but both fixes are verified present in the reviewed commit cd3b2a1 with substantive author replies: (1) the greedy retry.*$cmd wrapped-count regex was replaced with the anchored retry[[:space:]]+(sudo[[:space:]]+)?$cmd (correctly rejects cross-command spans and counts repeated commands), and (2) all three grep absence checks now have explicit exit-status-2 branches so a grep error hard-fails (exit 2) instead of silently passing. These are advisory-bot threads awaiting only the mechanical resolve click — no outstanding substance.
• All earlier advisory feedback (codeant-ai per-occurrence wrapping, full X.Y.Z pin, gemini normalization/[[:space:]] portability/case-insensitive pin check, qodo) is incorporated in HEAD.
• Workflow security review: no new actions, no uses: changes, no permissions changes, no untrusted-input interpolation in the new run: block; retry helper is plain bash over fixed commands. actionlint, CodeQL (actions), agent-shield, and Agent Security Scan all green.
• Secret scan: run_secret_scanning MCP tool not available in this environment; gitleaks CI check passed (no fabricated MCP result).
• Prior rate-limited marker (reset 2026-07-23T10:47:13Z) has elapsed; SonarCloud quality gate passed and advisory coverage (CodeAnt, Gemini, Qodo, Graphite, Sonar) is substantial despite Codex/CodeRabbit limits.

CI status

All required checks green at cd3b2a1: shellcheck/ShellCheck, actionlint, unit-tests, bats, Lint, CodeQL (actions + python), agent-shield, Agent Security Scan, gitleaks secret scan, SonarCloud quality gate, holdout-guard, caller-stub-freeze, template-drift, and all validate-* checks SUCCESS; remaining checks SKIPPED (not applicable). No failures.

---

Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[don-petry]

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning

  2. Grep error handling (lines 50-59, 88-92, 98-102): Added explicit exit status 2 checks to catch grep errors instead of silently passing
- ⚠️ **template-drift CI failure** (pre-existing): 7 workflow files show drift from standards-derived baseline:
  - `.github/workflows/agent-shield.yml`, `auto-rebase.yml`, `dependabot-automerge.yml`, `dependabot-rebase.yml`, `dependency-audit.yml`, `dev-lead.yml`, `pr-review-mention.yml`
  - **Root cause:** Merge commit d692255 from main updated these files, but `petry-projects/repo-template` has not been reseeded
  - **Resolution:** Requires separate action to run `scripts/seed-repo-template.sh` targeting repo-template (outside this PR's scope)
  - **Impact on PR:** This PR's changes to test.yml and test_test_workflow_resilience.sh are complete and correct; template-drift is a pre-merge condition
**Files Changed:** 
- `.github/workflows/test.yml` - Added retry helper and hardened dependency installation
- `tests/test_test_workflow_resilience.sh` - New regression guard (all checks passing)
**Status:** PR content is solid (SonarCloud ✅, graphite-app threads ✅, other checks ✅). Template-drift blocker requires coordinated action with repo-template repo ownership.