Add benchmark evidence reporter and automation#11
Conversation
Introduce GauntletCI.BenchmarkReporter, a .NET console app that generates benchmark evidence reports (JSON/CSV) from the curated fixture corpus. Add supporting test project for CLI parsing. Integrate a new GitHub Actions workflow to automate report generation and artifact upload. Update README with experimental evidence documentation and usage. No changes to core analysis logic; all additions focus on evidence reporting, automation, and documentation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df0891d83a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -1,5 +1,6 @@ | |||
| <Solution> | |||
| <Folder Name="/src/"> | |||
| <Project Path="src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj" /> | |||
There was a problem hiding this comment.
Include benchmark reporter tests in solution
Add tests/GauntletCI.BenchmarkReporter.Tests/GauntletCI.BenchmarkReporter.Tests.csproj to GauntletCI.slnx; otherwise the new parser tests are never executed in CI. I checked .github/workflows/ci.yml and .github/workflows/release.yml, and both run dotnet test GauntletCI.slnx, so omitting this project from the solution means regressions in the new reporter CLI parsing can ship unnoticed.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR introduces a new benchmark evidence reporting console app and associated automation to generate and publish JSON/CSV reports derived from the curated benchmark fixture corpus.
Changes:
- Add
GauntletCI.BenchmarkReporterconsole app to evaluate curated fixtures and emitlatest.json/latest.csv. - Add a new test project to validate CLI option parsing for the reporter.
- Add a scheduled/manual GitHub Actions workflow to generate and upload benchmark evidence artifacts; document usage in
README.md.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/GauntletCI.BenchmarkReporter/Program.cs |
Implements reporter CLI parsing, fixture evaluation, JSON/CSV output, and report schema records. |
src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj |
Adds new console app project and InternalsVisibleTo for tests. |
tests/GauntletCI.BenchmarkReporter.Tests/ReporterOptionsParseTests.cs |
Adds unit tests for reporter option parsing defaults and flags. |
tests/GauntletCI.BenchmarkReporter.Tests/GauntletCI.BenchmarkReporter.Tests.csproj |
Adds new xUnit test project referencing the reporter project. |
.github/workflows/benchmark-evidence.yml |
Automates report generation and uploads JSON/CSV artifacts on schedule/dispatch. |
docs/benchmarks/latest.json |
Adds generated benchmark evidence (JSON) snapshot. |
docs/benchmarks/latest.csv |
Adds generated benchmark evidence (CSV) snapshot. |
README.md |
Documents experimental evidence generation, outputs, and workflow automation. |
GauntletCI.slnx |
Adds the new reporter project to the solution. |
Comments suppressed due to low confidence (1)
GauntletCI.slnx:13
GauntletCI.slnxis used by CI (dotnet test GauntletCI.slnx) but the newtests/GauntletCI.BenchmarkReporter.Testsproject isn’t included under the/tests/folder. As a result, the new CLI parsing tests won’t run in CI or when running solution-based tests locally. Add the new test.csprojto the solution’s/tests/folder (and consider keeping the folder list complete for new test projects going forward).
<Solution>
<Folder Name="/src/">
<Project Path="src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj" />
<Project Path="src/GauntletCI.Cli/GauntletCI.Cli.csproj" />
<Project Path="src/GauntletCI.Core/GauntletCI.Core.csproj" />
<Project Path="src/GauntletCI.PrIntegration.Host/GauntletCI.PrIntegration.Host.csproj" />
<Project Path="src/GauntletCI.PrIntegration/GauntletCI.PrIntegration.csproj" />
</Folder>
<Folder Name="/tests/">
<Project Path="tests/GauntletCI.Benchmarks/GauntletCI.Benchmarks.csproj" />
<Project Path="tests/GauntletCI.Cli.Tests/GauntletCI.Cli.Tests.csproj" />
<Project Path="tests/GauntletCI.Core.Tests/GauntletCI.Core.Tests.csproj" />
</Folder>
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Usage: | ||
| dotnet run --project src/GauntletCI.BenchmarkReporter -- [options] | ||
|
|
||
| Options: | ||
| --repo-root <path> Repository root (default: current directory) | ||
| --fixtures-root <path> Curated fixtures root (default: ../tests/GauntletCI.Benchmarks/Fixtures/curated) | ||
| --output-dir <path> Output directory (default: docs/benchmarks) | ||
| --include-synthetic Include synthetic fixtures in metrics (default: false) | ||
| --help, -h Show help | ||
| """; |
There was a problem hiding this comment.
The help text defaults don’t match the actual parsing logic:
--repo-roothelp says “default: current directory”, butResolveDefaultRepoRoot()walks up to find a directory that looks like the repo root.--fixtures-roothelp says “default: ../tests/...”, butParse()defaults to<repoRoot>/tests/GauntletCI.Benchmarks/Fixtures/curated.
UpdateHelpTextso the documented defaults match the behavior.
| using System.Diagnostics; | ||
| using System.Globalization; | ||
| using System.Text; | ||
| using System.Text.Json; | ||
| using System.Text.Json.Serialization; | ||
| using GauntletCI.Core.Evaluation; | ||
| using GauntletCI.Core.Models; | ||
|
|
There was a problem hiding this comment.
New src/ C# source files in this repo consistently start with the SPDX license header (e.g., src/GauntletCI.Cli/Program.cs). GauntletCI.BenchmarkReporter/Program.cs currently lacks that header, which can create licensing/compliance inconsistencies. Add the standard SPDX + copyright header at the top of this file.
| if [ "${{ github.event.inputs.include_synthetic || 'false' }}" = "true" ]; then | ||
| args+=("--include-synthetic") | ||
| fi | ||
| dotnet run --project src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj --configuration Release -- "${args[@]}" |
There was a problem hiding this comment.
The workflow builds the reporter in Release (dotnet build ... --no-restore) and then runs it via dotnet run, which will trigger another build unless --no-build is specified. Consider adding --no-build to the dotnet run invocation to avoid redundant compilation and speed up scheduled runs.
| dotnet run --project src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj --configuration Release -- "${args[@]}" | |
| dotnet run --project src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj --configuration Release --no-build -- "${args[@]}" |
- Add GauntletCI.BenchmarkReporter.Tests to GauntletCI.slnx so CI picks up the reporter parser tests (was silently excluded before) - Fix --repo-root and --fixtures-root help text to match actual behavior (walks up from CWD; fixtures default is repo-relative) - Add SPDX license header to BenchmarkReporter/Program.cs for consistency with all other src/ files - Add --no-build to dotnet run in benchmark-evidence.yml to avoid redundant recompilation after the explicit dotnet build step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add GauntletCI.BenchmarkReporter.Tests to GauntletCI.slnx so CI picks up the reporter parser tests (was silently excluded before) - Fix --repo-root and --fixtures-root help text to match actual behavior (walks up from CWD; fixtures default is repo-relative) - Add SPDX license header to BenchmarkReporter/Program.cs for consistency with all other src/ files - Add --no-build to dotnet run in benchmark-evidence.yml to avoid redundant recompilation after the explicit dotnet build step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add 35 new deterministic signals from master rules list Extend DeterministicAnalysisRunner with two new capabilities: - RemovedLineRegex signal kind: scans removed diff lines (-) using the same pattern engine, enabling detection of safety regressions. - IncludedFilePathPatterns filter: optional allowlist so signals can be scoped to test files only (e.g. [Ignore]/[Skip], try-catch in tests). New AddedLine signals (17): DET-CONC-003 [ThreadStatic] attribute added (GCI016) DET-DISP-002 GC.SuppressFinalize without Dispose (GCI007) DET-PERF-004 MethodImplOptions.NoInlining added (GCI011) DET-API-004 sealed class/record added (GCI004) DET-TEST-002 [Ignore]/[Skip] in test files (GCI005) DET-TEST-003 try-catch inside test method (GCI005) DET-SEC-009 [SuppressUnmanagedCodeSecurity] added (GCI012) DET-SEC-011 [AllowAnonymous] added (GCI012) DET-SEC-012 http:// URL replacing https:// (GCI012) DET-SEC-013 Process.Start introduced (GCI012) DET-QUAL-001 [SuppressMessage] added (GCI013) DET-QUAL-002 #pragma warning disable added (GCI013) DET-QUAL-004 dynamic keyword introduced (GCI008) DET-QUAL-005 goto statement introduced (GCI008) DET-QUAL-006 [CLSCompliant(false)] added (GCI004) DET-INFRA-002 Timeout.Infinite introduced (GCI016) DET-INFRA-004 Environment.Exit/FailFast introduced (GCI003) New RemovedLine signals (18): DET-CONC-004 lock() removed (GCI016) DET-CONC-005 Parallel.ForEach removed (GCI016) DET-CONC-006 ConcurrentDictionary/Bag/Queue/Stack removed (GCI016) DET-CONC-007 ImmutableArray/List/Dictionary etc. removed (GCI016) DET-CONC-008 volatile keyword removed (GCI016) DET-INFRA-003 CancellationToken removed (GCI016) DET-MUTAT-001 readonly keyword removed (GCI016) DET-REL-004 finally block removed (GCI007) DET-REL-005 retry/Polly policy removed (GCI007) DET-INFRA-001 circuit breaker removed (GCI007) DET-DISP-001 using statement removed (GCI007) DET-PERF-003 ArrayPool usage removed (GCI011) DET-API-002 public member removed (GCI004) DET-API-003 [Obsolete] attribute removed (GCI004) DET-TEST-001 [Fact]/[Test] method removed (GCI005, test files only) DET-SEC-008 [Authorize] attribute removed (GCI012) DET-SEC-010 [SecurityCritical] attribute removed (GCI012) DET-QUAL-003 #nullable enable removed (GCI006) Also fix IsCommentLike to not suppress C# preprocessor directives (#pragma, #nullable, #if, #region, etc.) as comment-like lines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #11 review comments - Add GauntletCI.BenchmarkReporter.Tests to GauntletCI.slnx so CI picks up the reporter parser tests (was silently excluded before) - Fix --repo-root and --fixtures-root help text to match actual behavior (walks up from CWD; fixtures default is repo-relative) - Add SPDX license header to BenchmarkReporter/Program.cs for consistency with all other src/ files - Add --no-build to dotnet run in benchmark-evidence.yml to avoid redundant recompilation after the explicit dotnet build step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #12 bot review comments Fix 6 issues flagged by Copilot and Codex reviewers: 1. Spec-file inclusion regex false positives (Codex P2) [Ss]pec[^/]*\.cs$ matched Inspector.cs, Aspect.cs etc. because [^/]* allows arbitrary chars between 'spec' and '.cs'. Changed to (^|/)[^/]*[Ss]pecs?\.cs$ so only basenames ending in Spec/Specs are treated as test files. Applied to both AddedLine and RemovedLine spec builders. 2. Localhost exemption too broad in DET-SEC-012 (Codex P2) localhost\b treated localhost.example.com as loopback because \b fires on the t/. boundary. Changed to localhost(?![.\w]) so the exemption only fires when localhost is not followed by a dot or word character. 3. ParseFilePath breaks on paths with spaces (Copilot) Replaced space-split heuristic with a regex using a backreference (diff --git a/(.+) b/\1) that correctly identifies the split point regardless of spaces in paths. Falls back to LastIndexOf(' b/') for renames/copies where the two paths differ. 4. \ No newline at end of file advances line counter (Copilot) Lines starting with \ are git meta-markers, not real content. Added an early continue so they no longer shift subsequent finding line numbers. 5. actions/upload-artifact blocked by missing permission (Copilot) Added actions: write to the benchmark-evidence.yml permissions block so artifact upload does not fail under the restricted token. 6. --fixtures-root / --output-dir resolve against CWD not repo-root (Copilot) Deferred Path.GetFullPath resolution until after all args are parsed, then resolve relative paths against repoRoot instead of the current working directory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce GauntletCI.BenchmarkReporter, a .NET console app that generates benchmark evidence reports (JSON/CSV) from the curated fixture corpus. Add supporting test project for CLI parsing. Integrate a new GitHub Actions workflow to automate report generation and artifact upload. Update README with experimental evidence documentation and usage. No changes to core analysis logic; all additions focus on evidence reporting, automation, and documentation.