ci: add a manual workflow for the frame-transaction measurement harnesses

[AnkushinDaniil]

Re-bases #12738 onto eip8141-frame-txs-devnet7 where the frame-tx measurement harnesses live. Same workflow file, no content change; the five commits collapse to the single 103-line workflow. Closes #12738 (was based on master).

[claude]

Claude finished @AnkushinDaniil's task in 4m 43s —— View job

---

Review: ci: add a manual workflow for the frame-transaction measurement harnesses

• Gather context (diff vs origin/eip8141-frame-txs-devnet7, .agents/rules/github-workflows.md, the harnesses on this branch)
• Review the new workflow for correctness / security / conventions
• Post findings (5 inline comments)

Verdict: not mergeable as-is. The shape of the workflow is good — workflow_dispatch-only, permissions: contents: read, input passed through env: rather than interpolated into the script, action versions matching the repo's prevailing pins (checkout@v6, setup-dotnet@v5, upload-artifact@v7), the reproducible-benchmarks label spelled the same way as in run-expb-reproducible-benchmarks.yml and collect-pgo-profile.yml, no tracked file mutated. No security concerns and no hot-path impact. But the workflow cannot currently produce a measurement: it names a fixture that does not exist, and where it does name one it greps for figures on a stream the harnesses deliberately do not write to.

---

Critical

1. FrameTxMempoolDosMeasurement does not exist on this branch — line 75

The TxPool harness is Nethermind.TxPool.Test/FrameTxPrefixRetryMeasurement.cs. Since both is the default input, the presence probe trips on every default dispatch and the run ends red with ::error::FrameTxMempoolDosMeasurement is not present on <ref> — the very message that was meant to catch a wrong ref instead fires on a wrong name. Fix this →

High

2. The harnesses write RESULT lines to files, not stdout, so the summary and artifact stay empty — line 91

FrameTxPrefixRetryMeasurement.Emit appends to FRAME_RETRY_OUT (default Path.GetTempPath()/frame-prefix-retry.txt) and its <remarks> explains why: "the test runner swallows console writers". FrameTxVerifyDosMeasurement writes TestContext.Out plus a hard-coded /tmp/frame-dos-results.txt. The workflow tees stdout and greps that — so grep 'RESULT' is likely to hit nothing for any harness, and the run uploads an artifact with no figures in it. Secondary hazard: AppendAllText into /tmp on a persistent self-hosted runner accumulates across dispatches, so those files conflate runs. Set FRAME_RETRY_OUT="${OUTPUT_DIR}/${fixture}-results.txt", truncate it before the run, and grep there. Fix this →

3. Both fixtures are [Explicit("measurement harness")] — confirm the filter actually runs them — line 62

These projects use the NUnit MTP runner (EnableNUnitRunner=true in src/Nethermind/tests.props), not the VSTest adapter, and NUnit skips [Explicit] tests unless the filter counts as explicit selection. It is plausible that -- --filter "FullyQualifiedName~…" selects nothing here. Zero tests ran is at least a non-zero exit rather than a silent green, but the workflow may never have executed a harness. I could not verify this — dotnet test is not runnable in this review sandbox, so please dispatch once on the branch (or run the command locally) and confirm.

Low

4. Concurrency comment says dispatches "queue" — lines 16–22. The deviation itself is right and worth documenting; a group holds only one pending run, so a third dispatch cancels the second rather than joining a queue.

5. Presence probe keys off file name, not type name — lines 54–59. find … -name "${fixture}.cs" couples to a naming convention the --filter does not care about; grep -rq "class ${fixture}" probes the thing that is actually matched. Also set -uo pipefail without -e lets a failing mkdir -p pass unnoticed (the unset-OUTPUT_DIR case is covered by set -u).

Coverage note

Nethermind.Blockchain.Test/FrameTxProducerRetryMeasurement.cs is a third harness on this branch and is not reachable from any input option — it measures the per-attempt re-execution count that the other two multiply out, so it is arguably the one you most want on a shared runner.

Nothing to flag on secrets, injection, or performance; the if: always() && env.OUTPUT_DIR != '' guards and the ${GITHUB_STEP_SUMMARY} fencing are handled carefully.
· branch daniil/frame-measurement-workflow-d7

[AnkushinDaniil]

Superseded: #12738 stays on master. The measurement workflow is a manual workflow_dispatch bound to the reproducible-benchmarks self-hosted runner, which lives on the default branch, so it belongs on master rather than the devnet7 feature branch.

[claude]

Critical — the fixture name does not exist on this branch, so every default dispatch fails.

There is no FrameTxMempoolDosMeasurement anywhere in the tree. The TxPool harness on eip8141-frame-txs-devnet7 is Nethermind.TxPool.Test/FrameTxPrefixRetryMeasurement.cs (there is also Nethermind.Blockchain.Test/FrameTxProducerRetryMeasurement.cs).

So the presence probe on line 56 trips, run_harness returns 1, and — because both is the default input — the out-of-the-box dispatch always ends with ::error::FrameTxMempoolDosMeasurement is not present on <ref> and a red run, even though the verify half succeeded.

Suggested change

	run_harness Nethermind.TxPool.Test FrameTxMempoolDosMeasurement || status=1
	run_harness Nethermind.TxPool.Test FrameTxPrefixRetryMeasurement || status=1

The mempool choice label on line 13 stays accurate, but consider whether producer (for FrameTxProducerRetryMeasurement, the third harness) belongs in the option list too — it is the one that measures the per-attempt re-execution count that the other two multiply out.

[claude]

High — the harnesses do not put their figures on stdout, so this grep will report no RESULT lines.

FrameTxPrefixRetryMeasurement.Emit (and the identical Emit in FrameTxProducerRetryMeasurement) writes the RESULT … line only to a file:

string path = Environment.GetEnvironmentVariable("FRAME_RETRY_OUT")
              ?? Path.Combine(Path.GetTempPath(), "frame-prefix-retry.txt");
File.AppendAllText(path, $"RESULT {line}{Environment.NewLine}");

and its own <remarks> says why: "because the test runner swallows console writers". FrameTxVerifyDosMeasurement does TestContext.Out.WriteLine(line) and appends to a hard-coded /tmp/frame-dos-results.txt — under the NUnit MTP runner TestContext.Out for a passing test is not echoed to the console either, so the teed ${fixture}.txt is likely to contain no RESULT at all for any harness.

Two consequences:

1. The step summary and the uploaded artifact carry no figures — the workflow runs the measurement and then throws the measurement away.
2. Path.GetTempPath() / /tmp on a persistent self-hosted runner plus AppendAllText means the files accumulate across dispatches, so anything read out of them later silently mixes runs.

Fix: point the harness at ${OUTPUT_DIR} and truncate first, e.g. in the run step

        env:
          HARNESS: ${{ inputs.harness }}

            local out="${OUTPUT_DIR}/${fixture}-results.txt"
            : > "${out}"
            FRAME_RETRY_OUT="${out}" dotnet test ...

and then grep 'RESULT' "${OUTPUT_DIR}/${fixture}"*.txt. FrameTxVerifyDosMeasurement's /tmp/frame-dos-results.txt is hard-coded, so either copy it into ${OUTPUT_DIR} or teach that harness to honour FRAME_RETRY_OUT like the other two.

[claude]

High (needs one local verification run) — both harness fixtures are [Explicit("measurement harness")].

NUnit does not run [Explicit] tests unless the filter counts as an explicit selection, and these projects use the NUnit MTP runner (EnableNUnitRunner=true in src/Nethermind/tests.props), whose filter handling is not the VSTest adapter's. Two ways this can go wrong, and both are invisible until someone dispatches the workflow:

• the runner rejects/ignores VSTest-style FullyQualifiedName~… syntax, or
• it accepts it but still treats the fixture as explicit-only and runs zero tests.

Zero tests ran is at least not silently green (MTP exits non-zero on that), so the failure mode is a red run rather than an empty report — but the workflow as written may never have executed a harness. Please dispatch it once on the branch (or run the exact command locally) and confirm the harness bodies actually execute; if they do not, the filter needs whatever explicit-run form the NUnit MTP runner requires.

I could not verify this myself — dotnet test is not runnable in this review sandbox.

[claude]

Low — the rationale is right, the word "queue" overstates what GitHub does.

Dropping github.ref from the group and keeping cancel-in-progress: false is the correct call for a single shared measurement runner, and documenting it against the repo default in .agents/rules/github-workflows.md is welcome. But a concurrency group holds one pending run, not a queue: with a run in progress, dispatch #2 waits and dispatch #3 cancels #2. Worth a word in the comment so nobody is surprised when a dispatch disappears.

[claude]

Low — the presence probe assumes file name == fixture name, and find is doing a job [[ -f ]] can do.

It happens to hold for these harnesses, but it couples the workflow to a file-naming convention rather than to the type name; a fixture moved into a differently-named file reports "not present on this ref" while the test is right there. A grep -rq "class ${fixture}\b" "${project}" probe keys off the thing the --filter actually matches, and costs the same.

Minor also: set -uo pipefail without -e means the mkdir -p "${OUTPUT_DIR}" on line 52 can fail unnoticed. set -u does abort the step if OUTPUT_DIR is unset, so the dangerous case is covered — but a mkdir that fails for any other reason (permissions on the shared runner) surfaces only later as a tee error.