Skip to content

Add benchmark evidence reporter and automation#11

Merged
EricCogen merged 1 commit intomainfrom
benchmarkreporter
Apr 8, 2026
Merged

Add benchmark evidence reporter and automation#11
EricCogen merged 1 commit intomainfrom
benchmarkreporter

Conversation

@EricCogen
Copy link
Copy Markdown
Owner

Introduce GauntletCI.BenchmarkReporter, a .NET console app that generates benchmark evidence reports (JSON/CSV) from the curated fixture corpus. Add supporting test project for CLI parsing. Integrate a new GitHub Actions workflow to automate report generation and artifact upload. Update README with experimental evidence documentation and usage. No changes to core analysis logic; all additions focus on evidence reporting, automation, and documentation.

Introduce GauntletCI.BenchmarkReporter, a .NET console app that generates benchmark evidence reports (JSON/CSV) from the curated fixture corpus. Add supporting test project for CLI parsing. Integrate a new GitHub Actions workflow to automate report generation and artifact upload. Update README with experimental evidence documentation and usage. No changes to core analysis logic; all additions focus on evidence reporting, automation, and documentation.
Copilot AI review requested due to automatic review settings April 8, 2026 02:09
@EricCogen EricCogen merged commit 22ca30c into main Apr 8, 2026
6 checks passed
@EricCogen EricCogen deleted the benchmarkreporter branch April 8, 2026 02:10
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df0891d83a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@@ -1,5 +1,6 @@
<Solution>
<Folder Name="/src/">
<Project Path="src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj" />
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include benchmark reporter tests in solution

Add tests/GauntletCI.BenchmarkReporter.Tests/GauntletCI.BenchmarkReporter.Tests.csproj to GauntletCI.slnx; otherwise the new parser tests are never executed in CI. I checked .github/workflows/ci.yml and .github/workflows/release.yml, and both run dotnet test GauntletCI.slnx, so omitting this project from the solution means regressions in the new reporter CLI parsing can ship unnoticed.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new benchmark evidence reporting console app and associated automation to generate and publish JSON/CSV reports derived from the curated benchmark fixture corpus.

Changes:

  • Add GauntletCI.BenchmarkReporter console app to evaluate curated fixtures and emit latest.json / latest.csv.
  • Add a new test project to validate CLI option parsing for the reporter.
  • Add a scheduled/manual GitHub Actions workflow to generate and upload benchmark evidence artifacts; document usage in README.md.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/GauntletCI.BenchmarkReporter/Program.cs Implements reporter CLI parsing, fixture evaluation, JSON/CSV output, and report schema records.
src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj Adds new console app project and InternalsVisibleTo for tests.
tests/GauntletCI.BenchmarkReporter.Tests/ReporterOptionsParseTests.cs Adds unit tests for reporter option parsing defaults and flags.
tests/GauntletCI.BenchmarkReporter.Tests/GauntletCI.BenchmarkReporter.Tests.csproj Adds new xUnit test project referencing the reporter project.
.github/workflows/benchmark-evidence.yml Automates report generation and uploads JSON/CSV artifacts on schedule/dispatch.
docs/benchmarks/latest.json Adds generated benchmark evidence (JSON) snapshot.
docs/benchmarks/latest.csv Adds generated benchmark evidence (CSV) snapshot.
README.md Documents experimental evidence generation, outputs, and workflow automation.
GauntletCI.slnx Adds the new reporter project to the solution.
Comments suppressed due to low confidence (1)

GauntletCI.slnx:13

  • GauntletCI.slnx is used by CI (dotnet test GauntletCI.slnx) but the new tests/GauntletCI.BenchmarkReporter.Tests project isn’t included under the /tests/ folder. As a result, the new CLI parsing tests won’t run in CI or when running solution-based tests locally. Add the new test .csproj to the solution’s /tests/ folder (and consider keeping the folder list complete for new test projects going forward).
<Solution>
  <Folder Name="/src/">
    <Project Path="src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj" />
    <Project Path="src/GauntletCI.Cli/GauntletCI.Cli.csproj" />
    <Project Path="src/GauntletCI.Core/GauntletCI.Core.csproj" />
    <Project Path="src/GauntletCI.PrIntegration.Host/GauntletCI.PrIntegration.Host.csproj" />
    <Project Path="src/GauntletCI.PrIntegration/GauntletCI.PrIntegration.csproj" />
  </Folder>
  <Folder Name="/tests/">
    <Project Path="tests/GauntletCI.Benchmarks/GauntletCI.Benchmarks.csproj" />
    <Project Path="tests/GauntletCI.Cli.Tests/GauntletCI.Cli.Tests.csproj" />
    <Project Path="tests/GauntletCI.Core.Tests/GauntletCI.Core.Tests.csproj" />
  </Folder>

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +70 to +79
Usage:
dotnet run --project src/GauntletCI.BenchmarkReporter -- [options]

Options:
--repo-root <path> Repository root (default: current directory)
--fixtures-root <path> Curated fixtures root (default: ../tests/GauntletCI.Benchmarks/Fixtures/curated)
--output-dir <path> Output directory (default: docs/benchmarks)
--include-synthetic Include synthetic fixtures in metrics (default: false)
--help, -h Show help
""";
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The help text defaults don’t match the actual parsing logic:

  • --repo-root help says “default: current directory”, but ResolveDefaultRepoRoot() walks up to find a directory that looks like the repo root.
  • --fixtures-root help says “default: ../tests/...”, but Parse() defaults to <repoRoot>/tests/GauntletCI.Benchmarks/Fixtures/curated.
    Update HelpText so the documented defaults match the behavior.

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +8
using System.Diagnostics;
using System.Globalization;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
using GauntletCI.Core.Evaluation;
using GauntletCI.Core.Models;

Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New src/ C# source files in this repo consistently start with the SPDX license header (e.g., src/GauntletCI.Cli/Program.cs). GauntletCI.BenchmarkReporter/Program.cs currently lacks that header, which can create licensing/compliance inconsistencies. Add the standard SPDX + copyright header at the top of this file.

Copilot uses AI. Check for mistakes.
if [ "${{ github.event.inputs.include_synthetic || 'false' }}" = "true" ]; then
args+=("--include-synthetic")
fi
dotnet run --project src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj --configuration Release -- "${args[@]}"
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow builds the reporter in Release (dotnet build ... --no-restore) and then runs it via dotnet run, which will trigger another build unless --no-build is specified. Consider adding --no-build to the dotnet run invocation to avoid redundant compilation and speed up scheduled runs.

Suggested change
dotnet run --project src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj --configuration Release -- "${args[@]}"
dotnet run --project src/GauntletCI.BenchmarkReporter/GauntletCI.BenchmarkReporter.csproj --configuration Release --no-build -- "${args[@]}"

Copilot uses AI. Check for mistakes.
EricCogen added a commit that referenced this pull request Apr 8, 2026
- Add GauntletCI.BenchmarkReporter.Tests to GauntletCI.slnx so CI
  picks up the reporter parser tests (was silently excluded before)
- Fix --repo-root and --fixtures-root help text to match actual
  behavior (walks up from CWD; fixtures default is repo-relative)
- Add SPDX license header to BenchmarkReporter/Program.cs for
  consistency with all other src/ files
- Add --no-build to dotnet run in benchmark-evidence.yml to avoid
  redundant recompilation after the explicit dotnet build step

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EricCogen added a commit that referenced this pull request Apr 8, 2026
- Add GauntletCI.BenchmarkReporter.Tests to GauntletCI.slnx so CI
  picks up the reporter parser tests (was silently excluded before)
- Fix --repo-root and --fixtures-root help text to match actual
  behavior (walks up from CWD; fixtures default is repo-relative)
- Add SPDX license header to BenchmarkReporter/Program.cs for
  consistency with all other src/ files
- Add --no-build to dotnet run in benchmark-evidence.yml to avoid
  redundant recompilation after the explicit dotnet build step

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EricCogen added a commit that referenced this pull request Apr 8, 2026
* Add 35 new deterministic signals from master rules list

Extend DeterministicAnalysisRunner with two new capabilities:
- RemovedLineRegex signal kind: scans removed diff lines (-) using the
  same pattern engine, enabling detection of safety regressions.
- IncludedFilePathPatterns filter: optional allowlist so signals can be
  scoped to test files only (e.g. [Ignore]/[Skip], try-catch in tests).

New AddedLine signals (17):
  DET-CONC-003  [ThreadStatic] attribute added (GCI016)
  DET-DISP-002  GC.SuppressFinalize without Dispose (GCI007)
  DET-PERF-004  MethodImplOptions.NoInlining added (GCI011)
  DET-API-004   sealed class/record added (GCI004)
  DET-TEST-002  [Ignore]/[Skip] in test files (GCI005)
  DET-TEST-003  try-catch inside test method (GCI005)
  DET-SEC-009   [SuppressUnmanagedCodeSecurity] added (GCI012)
  DET-SEC-011   [AllowAnonymous] added (GCI012)
  DET-SEC-012   http:// URL replacing https:// (GCI012)
  DET-SEC-013   Process.Start introduced (GCI012)
  DET-QUAL-001  [SuppressMessage] added (GCI013)
  DET-QUAL-002  #pragma warning disable added (GCI013)
  DET-QUAL-004  dynamic keyword introduced (GCI008)
  DET-QUAL-005  goto statement introduced (GCI008)
  DET-QUAL-006  [CLSCompliant(false)] added (GCI004)
  DET-INFRA-002 Timeout.Infinite introduced (GCI016)
  DET-INFRA-004 Environment.Exit/FailFast introduced (GCI003)

New RemovedLine signals (18):
  DET-CONC-004  lock() removed (GCI016)
  DET-CONC-005  Parallel.ForEach removed (GCI016)
  DET-CONC-006  ConcurrentDictionary/Bag/Queue/Stack removed (GCI016)
  DET-CONC-007  ImmutableArray/List/Dictionary etc. removed (GCI016)
  DET-CONC-008  volatile keyword removed (GCI016)
  DET-INFRA-003 CancellationToken removed (GCI016)
  DET-MUTAT-001 readonly keyword removed (GCI016)
  DET-REL-004   finally block removed (GCI007)
  DET-REL-005   retry/Polly policy removed (GCI007)
  DET-INFRA-001 circuit breaker removed (GCI007)
  DET-DISP-001  using statement removed (GCI007)
  DET-PERF-003  ArrayPool usage removed (GCI011)
  DET-API-002   public member removed (GCI004)
  DET-API-003   [Obsolete] attribute removed (GCI004)
  DET-TEST-001  [Fact]/[Test] method removed (GCI005, test files only)
  DET-SEC-008   [Authorize] attribute removed (GCI012)
  DET-SEC-010   [SecurityCritical] attribute removed (GCI012)
  DET-QUAL-003  #nullable enable removed (GCI006)

Also fix IsCommentLike to not suppress C# preprocessor directives
(#pragma, #nullable, #if, #region, etc.) as comment-like lines.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #11 review comments

- Add GauntletCI.BenchmarkReporter.Tests to GauntletCI.slnx so CI
  picks up the reporter parser tests (was silently excluded before)
- Fix --repo-root and --fixtures-root help text to match actual
  behavior (walks up from CWD; fixtures default is repo-relative)
- Add SPDX license header to BenchmarkReporter/Program.cs for
  consistency with all other src/ files
- Add --no-build to dotnet run in benchmark-evidence.yml to avoid
  redundant recompilation after the explicit dotnet build step

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #12 bot review comments

Fix 6 issues flagged by Copilot and Codex reviewers:

1. Spec-file inclusion regex false positives (Codex P2)
   [Ss]pec[^/]*\.cs$ matched Inspector.cs, Aspect.cs etc. because
   [^/]* allows arbitrary chars between 'spec' and '.cs'. Changed to
   (^|/)[^/]*[Ss]pecs?\.cs$ so only basenames ending in Spec/Specs
   are treated as test files. Applied to both AddedLine and
   RemovedLine spec builders.

2. Localhost exemption too broad in DET-SEC-012 (Codex P2)
   localhost\b treated localhost.example.com as loopback because \b
   fires on the t/. boundary. Changed to localhost(?![.\w]) so the
   exemption only fires when localhost is not followed by a dot or
   word character.

3. ParseFilePath breaks on paths with spaces (Copilot)
   Replaced space-split heuristic with a regex using a backreference
   (diff --git a/(.+) b/\1) that correctly identifies the split point
   regardless of spaces in paths. Falls back to LastIndexOf(' b/')
   for renames/copies where the two paths differ.

4. \ No newline at end of file advances line counter (Copilot)
   Lines starting with \ are git meta-markers, not real content.
   Added an early continue so they no longer shift subsequent finding
   line numbers.

5. actions/upload-artifact blocked by missing permission (Copilot)
   Added actions: write to the benchmark-evidence.yml permissions
   block so artifact upload does not fail under the restricted token.

6. --fixtures-root / --output-dir resolve against CWD not repo-root (Copilot)
   Deferred Path.GetFullPath resolution until after all args are
   parsed, then resolve relative paths against repoRoot instead of
   the current working directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants