Add runner name as metric label and benchmark name param #4243

aaronbuchwald · 2025-09-04T16:00:21Z

Why this should be merged

This PR adds a runnerName argument to the benchmark tests, which is attached as both a metric label and incorporated into the benchmark name. This ensures that runs from two different machine types will not be treated the same and we know to differentiate runs on ARC from runs on GitHub runners, blacksmith, local runs, etc.

How this works

Adds a runnerName flag to the benchmark, passes it through Taskfile and the benchmark script, and adds it into the custom action and workflow triggers.

How this was tested

Testing locally shows that the default value of runner is correctly attached to the metrics and shows up in the benchmark name:

Metrics from local run here: https://grafana-poc.avax-dev.network/d/Gl1I20mnk/c-chain?orgId=1&from=now-15m&to=now&timezone=America%2FNew_York&var-datasource=P1809F7CD0C75ACF3&var-filter=runner%7C%3D%7Cdev&var-chain=C&refresh=10s

Output:

[09-04|11:55:21.810] INFO c-chain-reexecution c/vm_reexecute_test.go:206 shutting down DB
BenchmarkReexecuteRange/[101,5000]-Config-default-Runner-dev-12                        1               258.2 mgas/s
PASS
ok      github.com/ava-labs/avalanchego/tests/reexecute/c       16.453s

From CI:

Metrics correctly attach the new label: https://grafana-poc.avax-dev.network/d/Gl1I20mnk/c-chain?orgId=1&refresh=10s&var-filter=runner%7C%3D%7Cblacksmith-4vcpu-ubuntu-2404&from=2025-09-04T16:14:31.000Z&to=2025-09-04T16:29:31.000Z&timezone=America%2FNew_York&var-datasource=P1809F7CD0C75ACF3&var-chain=C

Output:

[09-04|16:18:52.233] INFO <2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5 Chain> eth/backend.go:409 Stopped EventMux
[09-04|16:18:52.233] INFO c-chain-reexecution c/vm_reexecute_test.go:206 shutting down DB
BenchmarkReexecuteRange/[101,250000]-Config-default-Runner-blacksmith-4vcpu-ubuntu-2404-6         	       1	       103.0 mgas/s
PASS
ok  	github.com/ava-labs/avalanchego/tests/reexecute/c	261.726s

Need to be documented in RELEASES.md?

No

Copilot

Pull Request Overview

This PR adds support for differentiating benchmark runs by adding a runner name parameter that is included in both metric labels and benchmark names, enabling identification of results from different execution environments (ARC, GitHub runners, Blacksmith, local runs, etc.).

Adds runnerNameArg flag to benchmark tests with default value "dev"
Includes runner name in benchmark sub-test names and as a metric label
Passes runner name through the entire CI/CD pipeline from workflows to scripts

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/reexecute/c/vm_reexecute_test.go	Adds runner name flag, metric label, and incorporates it into benchmark naming
scripts/benchmark_cchain_range.sh	Adds RUNNER_NAME environment variable requirement and passes it to test command
Taskfile.yml	Adds RUNNER_NAME variable with "dev" default to benchmark tasks
.github/workflows/c-chain-reexecution-benchmark-gh-native.yml	Passes matrix runner name to custom action
.github/workflows/c-chain-reexecution-benchmark-container.yml	Passes matrix runner name to custom action
.github/actions/c-chain-reexecution-benchmark/action.yml	Adds runner_name input and passes it through environment variables

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/reexecute/c/vm_reexecute_test.go

maru-ava · 2025-09-04T20:27:03Z

Taskfile.yml

          LABELS: '{{.LABELS}}'
          BENCHMARK_OUTPUT_FILE: '{{.BENCHMARK_OUTPUT_FILE}}'
          METRICS_ENABLED: '{{.METRICS_ENABLED}}'
+


Superfluous whitespace?

maru-ava · 2025-09-04T20:30:05Z

tests/reexecute/c/vm_reexecute_test.go

-	b.Run(fmt.Sprintf("[%d,%d]-Config-%s", startBlockArg, endBlockArg, configNameArg), func(b *testing.B) {
-		benchmarkReexecuteRange(b, blockDirArg, currentStateDirArg, configBytesArg, startBlockArg, endBlockArg, chanSizeArg, metricsEnabledArg)
+	b.Run(fmt.Sprintf("[%d,%d]-Config-%s-Runner-%s", startBlockArg, endBlockArg, configNameArg, runnerNameArg), func(b *testing.B) {
+		benchmarkReexecuteRange(


(No action required) Maybe avoid unnecessary formatting changes to things that aren't changing? Same comment for L151

maru-ava · 2025-09-04T20:30:53Z

tests/reexecute/c/vm_reexecute_test.go

-func parseLabels(labelsStr string) (map[string]string, error) {
+// parseCustomLabels parses a comma-separated list of key-value pairs into a map
+// of custom labels.
+func parseCustomLabels(labelsStr string) (map[string]string, error) {


(No action required) Maybe avoid including non-functional changes with functional ones?

yacovm · 2025-09-04T20:55:09Z

.github/actions/c-chain-reexecution-benchmark/action.yml

      with:
-        run: |
-          ./scripts/run_task.sh reexecute-cchain-range-with-copied-data \
+        run_env: |


why this change?

It was already a very long line, this made it longer, and I realized the run_env param supported this.

RodrigoVillar

Approving with the assumption that the (maybe?) cosmetic changes in this PR are not blockers for other reviewers.

Taskfile.yml

Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com>

aaronbuchwald added 3 commits September 4, 2025 11:32

Move task params to run_env input

11306c5

Add runner name to reexecute job and task

3d82a1b

fix passing runner param via taskfile

bf4e1c7

github-project-automation bot added this to avalanchego Sep 4, 2025

aaronbuchwald self-assigned this Sep 4, 2025

aaronbuchwald changed the title ~~Aaronbuchwald/add runner to benchmark name~~ Add runner name as metric label and benchmark name param Sep 4, 2025

aaronbuchwald added 3 commits September 4, 2025 12:10

pass runner flag correctly

5fbb496

fix flag

afaffd3

fix lint

7290f9d

aaronbuchwald marked this pull request as ready for review September 4, 2025 20:10

Copilot AI review requested due to automatic review settings September 4, 2025 20:10

aaronbuchwald requested review from joshua-kim and maru-ava as code owners September 4, 2025 20:10

aaronbuchwald enabled auto-merge September 4, 2025 20:10

aaronbuchwald requested a review from RodrigoVillar September 4, 2025 20:10

Copilot AI reviewed Sep 4, 2025

View reviewed changes

tests/reexecute/c/vm_reexecute_test.go Show resolved Hide resolved

aaronbuchwald requested review from samliok and yacovm September 4, 2025 20:11

maru-ava approved these changes Sep 4, 2025

View reviewed changes

yacovm reviewed Sep 4, 2025

View reviewed changes

RodrigoVillar approved these changes Sep 5, 2025

View reviewed changes

aaronbuchwald added this pull request to the merge queue Sep 5, 2025

RodrigoVillar removed this pull request from the merge queue due to a manual request Sep 5, 2025

aaronbuchwald commented Sep 5, 2025

View reviewed changes

Taskfile.yml Outdated Show resolved Hide resolved

Update Taskfile.yml

1414a33

Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com>

aaronbuchwald enabled auto-merge September 5, 2025 13:41

aaronbuchwald added this pull request to the merge queue Sep 5, 2025

Merged via the queue into master with commit 3983b46 Sep 5, 2025
35 checks passed

aaronbuchwald deleted the aaronbuchwald/add-runner-to-benchmark-name branch September 5, 2025 14:13

github-project-automation bot moved this to Done 🎉 in avalanchego Sep 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add runner name as metric label and benchmark name param #4243

Add runner name as metric label and benchmark name param #4243

Uh oh!

aaronbuchwald commented Sep 4, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

maru-ava Sep 4, 2025

Uh oh!

maru-ava Sep 4, 2025

Uh oh!

maru-ava Sep 4, 2025

Uh oh!

yacovm Sep 4, 2025

Uh oh!

aaronbuchwald Sep 5, 2025

Uh oh!

RodrigoVillar left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Add runner name as metric label and benchmark name param #4243

Add runner name as metric label and benchmark name param #4243

Uh oh!

Conversation

aaronbuchwald commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this works

How this was tested

Need to be documented in RELEASES.md?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

maru-ava Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

maru-ava Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

maru-ava Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

yacovm Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

aaronbuchwald Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

RodrigoVillar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aaronbuchwald commented Sep 4, 2025 •

edited

Loading