chore(ci): use self-hosted runners for benchmarks #61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Switch benchmark jobs to use self-hosted runners with 'benchmark-runner' label for consistent performance measurements. Changes: - .NET 8.0 job: ubuntu-latest → [self-hosted, Linux, benchmark-runner] - .NET Framework 4.7.2 job: windows-2022 → [self-hosted, Windows, benchmark-runner] Benefits: - Consistent hardware specs across benchmark runs - Predictable network performance to Databricks - Eliminates variability from shared GitHub-hosted runners Note: Requires self-hosted runners with 'benchmark-runner' label: - Linux runner for .NET 8.0 benchmarks - Windows runner for .NET Framework 4.7.2 benchmarks 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Keep .NET 8.0 job on GitHub-hosted ubuntu-latest since only Windows self-hosted runner is available. Changes: - .NET 8.0 job: stays on ubuntu-latest (GitHub-hosted) - .NET Framework 4.7.2 job: uses [self-hosted, Windows, benchmark-runner] 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
.NET 8.0 is cross-platform and runs on Windows, so both jobs can use the same Windows self-hosted runner. Changes: - .NET 8.0 job: ubuntu-latest → [self-hosted, Windows, benchmark-runner] - .NET Framework 4.7.2 job: [self-hosted, Windows, benchmark-runner] (unchanged) Both benchmark jobs now run on the same consistent Windows hardware. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Ensure benchmarks run one at a time to prevent parallel downloads from degrading performance measurements. Changes: - Updated concurrency group to queue all benchmark runs - Set cancel-in-progress: false (queue instead of cancel) - Made .NET Framework 4.7.2 job depend on .NET 8.0 completion This ensures: - Only one workflow runs at a time (no parallel workflow runs) - Jobs run sequentially (no parallel job execution) - Consistent network bandwidth for each benchmark - Accurate performance measurements 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Self-hosted runners should have .NET pre-installed to avoid permission issues when setup-dotnet tries to write to C:\Program Files\dotnet. Changes: - Added condition to skip setup-dotnet on self-hosted runners - Setup-dotnet only runs on GitHub-hosted runners now Requirements for self-hosted runner: - .NET 8.0 SDK must be pre-installed - .NET Framework 4.7.2 (usually pre-installed on Windows) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
The script uses bash syntax (heredoc, if [ -z ], etc.) but was running in PowerShell on Windows, causing syntax errors. Changes: - Added 'shell: bash' to .NET 8.0 config file creation step - .NET Framework 4.7.2 job already uses 'shell: pwsh' correctly Requires Git Bash to be available on Windows self-hosted runner (usually installed with Git for Windows). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Temporarily disable .NET 8.0 benchmark job due to environment issues. Only run .NET Framework 4.7.2 benchmarks on self-hosted Windows runner. Changes: - Set benchmark-net8 job condition to 'false' (disabled) - Removed dependency from benchmark-net472 job - .NET Framework 4.7.2 job runs independently now 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Change from 'pwsh' to 'powershell' since PowerShell Core is not installed on the self-hosted Windows runner. Windows PowerShell is built-in to Windows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Convert build and benchmark steps from bash scripts to inline PowerShell commands for compatibility with Windows self-hosted runner without Git Bash. Changes: - Build step: Direct dotnet build command in PowerShell - Benchmark step: Direct dotnet run command in PowerShell - All steps now use 'shell: powershell' 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…ndows" This reverts commit 0bfaed1.
Replace remaining PowerShell Core (pwsh) references with Windows PowerShell (powershell) for self-hosted Windows runner. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Use System.IO.File.WriteAllText with UTF8Encoding($false) to write JSON without BOM, fixing JSON.parse() errors in github-script action. PowerShell's Out-File -Encoding utf8 adds a BOM which causes: "Unexpected token ''" error in JavaScript JSON parsing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
eric-wang-1990
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
Re-enable .NET 8.0 job now that Git Bash is installed on the runner. Both jobs will run sequentially on the Windows self-hosted runner. Changes: - Re-enabled benchmark-net8 job - Added sequential dependency (net472 waits for net8) - Both jobs use Windows self-hosted runner with benchmark-runner label 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
The extract-data step uses bash syntax but was missing shell specification, causing it to default to PowerShell on Windows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
Convert extract-data and create-benchmark-data steps from bash/jq to PowerShell to avoid requiring jq installation on Windows runner. Changes: - Extract-data: Uses PowerShell ConvertFrom-Json instead of jq - Create-benchmark-data: Uses PowerShell with UTF-8 without BOM - Both jobs now consistently use PowerShell for JSON operations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
had a problem deploying
to
azure-prod
GitHub Actions
Failure
The download baseline step uses bash syntax (||, mkdir -p, node) but was missing shell specification, causing it to default to PowerShell. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
eric-wang-1990
added
benchmark
eric-wang-1990
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
🎯 Benchmark Results (.NET 8.0)
Indicators:
|
eric-wang-1990
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
🎯 Benchmark Results (.NET Framework 4.7.2)
Indicators:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Switch both benchmark jobs to use a self-hosted Windows runner and serialize execution to ensure consistent performance measurements without network contention.
Changes
Runner Configuration
ubuntu-latest→[self-hosted, Windows, benchmark-runner]windows-2022→[self-hosted, Windows, benchmark-runner]Execution Serialization
Benefits
✅ Consistent hardware specs - Same CPU, RAM, and disk for all benchmark runs
✅ No network contention - Only one benchmark downloads data at a time
✅ Predictable network performance - Full bandwidth available to each benchmark
✅ Accurate measurements - No degradation from parallel execution
✅ Better baseline comparisons - Apples-to-apples performance comparisons
✅ Single runner - Only need one Windows self-hosted runner for both jobs
How It Works
Workflow-level serialization:
Job-level serialization:
Requirements
A self-hosted Windows runner with the
benchmark-runnerlabel. Both .NET 8.0 and .NET Framework 4.7.2 benchmarks will run sequentially on this runner.Testing
After merging, verify the serialization works:
🤖 Generated with Claude Code