Update flaky-monitor workflow to show fail rate and history#1395
Merged
KCarretto merged 2 commits intoflaky-test-monitor-3414294539595760241from Dec 24, 2025
Conversation
Enable `fail-rate-report` and `previous-results-report` in the Github Test Reporter action configuration. Set the history limit to 5 runs to provide a concise view of recent sporadic failures.
Contributor
Author
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. New to Jules? Learn more at jules.google/docs. |
KCarretto
reviewed
Dec 24, 2025
.github/workflows/tests.yml
Outdated
| previous-results-max: 100 | ||
| fail-rate-report: true | ||
| previous-results-report: true | ||
| previous-results-max: 5 |
Contributor
Author
There was a problem hiding this comment.
I have set previous-results-max back to 100 as requested.
Reverted `previous-results-max` to 100 as requested in code review, while keeping `fail-rate-report` and `previous-results-report` enabled.
KCarretto
deleted the
chore/update-flaky-monitor-config-2580247471303683982
branch
KCarretto
added a commit
that referenced
this pull request
Dec 24, 2025
* Implement flaky test monitoring system - Update `tests.yml` to generate JUnit XML reports for Go, Rust, and Node.js tests. - Add `flaky-monitor` job to aggregate and analyze historical test results. - Create `scripts/analyze_flaky_tests.py` to calculate test flakiness grouped by OS. - Configure automatic reporting to GitHub Step Summary and PR comments. - Use `test-results` branch for persistent history storage. * Implement flaky test monitoring using CTRF - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by using `--` separator. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. - Fix PATH issues for `go-ctrf-json-reporter` by explicitly adding GOPATH/bin to GITHUB_PATH. - Ensure `npx` is available in Rust job by adding `setup-node` step. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. - Fix PATH issues for `go-ctrf-json-reporter` by calling binary with full path. - Ensure `npx` is available in Rust job by adding `setup-node` step. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. - Fix PATH issues for `go-ctrf-json-reporter` by calling binary with full path. - Ensure `npx` is available in Rust job by adding `setup-node` step. - Pin `gotestsum` to v1.12.0 to support Go 1.23.4. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. - Fix PATH issues for `go-ctrf-json-reporter` by calling binary with full path. - Ensure `npx` is available in Rust job by adding `setup-node` step. - Pin `gotestsum` to v1.12.0 to support Go 1.23.4. - Use `shell: bash` explicitly for steps using bash syntax to prevent PowerShell errors on Windows runners. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. - Fix PATH issues for `go-ctrf-json-reporter` by calling binary with full path. - Ensure `npx` is available in Rust job by adding `setup-node` step. - Pin `gotestsum` to v1.12.0 to support Go 1.23.4. - Use `shell: bash` explicitly for steps using bash syntax to prevent PowerShell errors on Windows runners. - Revert unintended version downgrade in `.devcontainer/devcontainer.json`. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. - Fix PATH issues for `go-ctrf-json-reporter` by calling binary with full path. - Ensure `npx` is available in Rust job by adding `setup-node` step. - Pin `gotestsum` to v1.12.0 to support Go 1.23.4. - Use `shell: bash` explicitly for steps using bash syntax to prevent PowerShell errors on Windows runners. - Revert unintended version downgrade in `.devcontainer/devcontainer.json`. - Restore missing documentation in `docs/_docs/user-guide/imix.md`. * Implement CTRF-based flaky test monitoring - Update `tests.yml` to generate CTRF reports for Go, Rust, and Node.js tests. - Use `go-ctrf-json-reporter` for Go tests. - Use `junit-to-ctrf` for Rust and Node.js tests (converting JUnit XML). - Add `flaky-monitor` job using `ctrf-io/github-test-reporter` to track and report flaky tests with comprehensive reporting options enabled. - Configure historical data tracking via artifact retention. - Fix argument passing for `cargo llvm-cov` by configuring `nextest.toml` for JUnit output. - Fix PATH issues for `go-ctrf-json-reporter` by calling binary with full path. - Ensure `npx` is available in Rust job by adding `setup-node` step. - Pin `gotestsum` to v1.12.0 to support Go 1.23.4. - Use `shell: bash` explicitly for steps using bash syntax to prevent PowerShell errors on Windows runners. - Revert unintended version downgrade in `.devcontainer/devcontainer.json`. - Restore missing documentation in `docs/_docs/user-guide/imix.md`. - Correctly locate `nextest.toml` in `.config/` directory. * switching to manual * attempt #2 * overwrite-comment * Fix CI workflow to correctly locate nested JUnit reports (#1388) The `implants` test job was failing to generate CTRF reports because the `junit.xml` output from `cargo nextest` was located in a subdirectory (e.g., `target/nextest/default/junit.xml`) rather than the workspace root. This caused the `mv` command to fail, skipping the conversion step and ultimately causing the artifact upload to fail with "No files were found". This change updates the workflow to dynamically search for `junit.xml` using `find` (piped to `head -n 1` for macOS compatibility), ensuring the report is correctly located and processed regardless of the output directory structure. This also restores the `implants/.config/nextest.toml` configuration to ensure JUnit generation is enabled. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * Fix: Normalize timestamps in Tavern CTRF reports (#1391) * Update flaky-monitor workflow to show fail rate and history (#1395) --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: KCarretto <Kcarretto@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated the
.github/workflows/tests.ymlfile to configure thectrf-io/github-test-reporteraction. Addedfail-rate-report: trueandprevious-results-report: trueoptions, and setprevious-results-maxto 5. This ensures that sporadically failing tests are tracked via their failure rate and a history of the last 5 runs is displayed in the test report.PR created automatically by Jules for task 2580247471303683982 started by @KCarretto