Skip to content

Conversation

@mcalhoun
Copy link
Contributor

@mcalhoun mcalhoun commented Oct 25, 2025

ci: parallelize benchmark workflow for faster execution

What this PR does / why we need it:

This PR enhances the benchmark workflow by parallelizing benchmark execution across packages. The changes include:

  1. Adding a new "discover" job that finds all packages containing benchmark tests
  2. Modifying the benchmark job to run in parallel using a matrix strategy based on discovered packages
  3. Implementing artifact uploads for individual benchmark results
  4. Adding a "report" job that combines all benchmark results into a single PR comment
  5. Adding new parameters to the bench/check make target:
    • count parameter (defaults to 20) to control benchmark iterations
    • parallel parameter (defaults to 1) to control Go test parallelism

These changes improve CI efficiency by running benchmarks for different packages concurrently rather than sequentially.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

The workflow now automatically discovers packages with benchmark tests and runs them in parallel, which should significantly reduce the overall benchmark execution time.

Does this PR introduce a user-facing change?

No


Note

Parallelizes Go benchmarks per-package with matrix jobs, uploads results as artifacts, aggregates them into a single PR comment, and adds count/parallel options to bench/check.

  • CI · benchmarks workflow (.github/workflows/benchmark.yml):
    • Add discover job to find packages containing benchmark functions.
    • Run benchmarks per-package via matrix; generate package-scoped result files and upload as artifacts.
    • Add report job to download, combine, and post aggregated benchmark results to the PR.
  • Makefile (makefiles/common.mk):
    • Enhance bench/check with count and parallel parameters (defaults: 20, 1).
    • Pass -parallel and dynamic -count through -go-test-flags.

Written by Cursor Bugbot for commit 9e486cf. This will update automatically on new commits. Configure here.

Copy link
Contributor Author

mcalhoun commented Oct 25, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@mcalhoun mcalhoun marked this pull request as ready for review October 25, 2025 15:48
@mcalhoun mcalhoun requested a review from a team as a code owner October 25, 2025 15:48
cursor[bot]

This comment was marked as outdated.

@mcalhoun mcalhoun force-pushed the chore/mcalhoun/speed-up-ci-jobs branch from 23d7acb to 4798329 Compare October 25, 2025 15:51
@github-actions
Copy link

github-actions bot commented Oct 25, 2025

## Package: benchmarks/changed
metric: time/op
ChangeDetection-6: old 4.29ms ±39%: new 5.43ms ±16%: delta: 26.57%
ChangeDetectionTFAndTG-6: old 4.17ms ±50%: new 5.02ms ±14%: delta: 20.26%
check failed: time/op=+20%
metric: alloc/op
ChangeDetection-6: old 356kB ± 0%: new 357kB ± 0%: delta: 0.03%
ChangeDetectionTFAndTG-6: old 261kB ± 5%: new 267kB ± 4%: delta: 2.49%
metric: allocs/op
ChangeDetection-6: old 2.41k ± 0%: new 2.41k ± 0%: delta: 0.02%
ChangeDetectionTFAndTG-6: old 1.99k ±15%: new 2.14k ±12%: delta: 7.65%

---

## Package: cloud
metric: time/op
CloudReadLines-6: old 944µs ±19%: new 975µs ±16%: delta: 0.00%
CloudReadLine-6: old 5.04ms ± 8%: new 5.35ms ± 9%: delta: 6.03%
metric: alloc/op
CloudReadLines-6: old 3.12MB ± 0%: new 3.12MB ± 0%: delta: -0.00%
CloudReadLine-6: old 3.37MB ± 0%: new 3.37MB ± 0%: delta: -0.00%
metric: allocs/op
CloudReadLines-6: old 5.54k ± 0%: new 5.54k ± 0%: delta: 0.00%
CloudReadLine-6: old 60.0k ± 0%: new 60.0k ± 0%: delta: 0.00%

---

## Package: fs
metric: time/op
ListFiles-6: old 49.3µs ±15%: new 52.1µs ±15%: delta: 0.00%
metric: alloc/op
ListFiles-6: old 27.6kB ± 0%: new 27.6kB ± 0%: delta: 0.00%
metric: allocs/op
ListFiles-6: old 335 ± 0%: new 335 ± 0%: delta: 0.00%

---

## Package: generate
metric: time/op
Generate-6: old 660ms ± 2%: new 666ms ± 4%: delta: 1.01%
GenerateRegex-6: old 418ms ± 2%: new 453ms ±11%: delta: 8.39%
metric: alloc/op
Generate-6: old 2.23GB ± 0%: new 2.23GB ± 0%: delta: 0.00%
GenerateRegex-6: old 923MB ± 0%: new 923MB ± 0%: delta: 0.00%
metric: allocs/op
Generate-6: old 25.7M ± 0%: new 25.7M ± 0%: delta: 0.00%
GenerateRegex-6: old 18.1M ± 0%: new 18.1M ± 0%: delta: 0.00%

---

## Package: hcl/ast
metric: time/op
TokensForExpressionComplex-6: old 1.34ms ±20%: new 1.04ms ± 6%: delta: -22.72%
TokensForExpressionPlainStringNoNewline-6: old 698ns ±23%: new 536ns ± 5%: delta: -23.29%
TokensForExpressionStringWith100Newlines-6: old 21.2µs ±19%: new 14.6µs ±10%: delta: -31.17%
TokensForExpressionObjectWith100KeysWithNumberValues-6: old 1.24ms ± 7%: new 1.22ms ± 2%: delta: 0.00%
TokensForExpression-6: old 1.12ms ±11%: new 1.09ms ± 6%: delta: 0.00%
metric: alloc/op
TokensForExpressionComplex-6: old 393kB ± 0%: new 393kB ± 0%: delta: -0.05%
TokensForExpressionPlainStringNoNewline-6: old 512B ± 0%: new 512B ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-6: old 12.5kB ± 0%: new 12.5kB ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-6: old 393kB ± 0%: new 393kB ± 0%: delta: 0.00%
TokensForExpression-6: old 393kB ± 0%: new 393kB ± 0%: delta: 0.01%
metric: allocs/op
TokensForExpressionComplex-6: old 4.75k ± 0%: new 4.75k ± 0%: delta: -0.01%
TokensForExpressionPlainStringNoNewline-6: old 20.0 ± 0%: new 20.0 ± 0%: delta: 0.00%
TokensForExpressionStringWith100Newlines-6: old 227 ± 0%: new 227 ± 0%: delta: 0.00%
TokensForExpressionObjectWith100KeysWithNumberValues-6: old 3.09k ± 0%: new 3.09k ± 0%: delta: 0.00%
TokensForExpression-6: old 4.75k ± 0%: new 4.75k ± 0%: delta: 0.00%

---

## Package: hcl/eval
metric: time/op
PartialEvalComplex-6: old 449µs ±18%: new 439µs ±24%: delta: 0.00%
PartialEvalSmallString-6: old 3.36µs ±16%: new 3.32µs ±13%: delta: 0.00%
PartialEvalHugeString-6: old 1.47ms ± 1%: new 1.46ms ± 2%: delta: -0.79%
PartialEvalHugeInterpolatedString-6: old 4.48ms ±11%: new 4.56ms ± 9%: delta: 0.00%
PartialEvalObject-6: old 20.1µs ± 7%: new 25.4µs ±16%: delta: 25.91%
check failed: time/op=+20%
metric: alloc/op
PartialEvalComplex-6: old 360kB ± 0%: new 360kB ± 0%: delta: 0.00%
PartialEvalSmallString-6: old 1.94kB ± 0%: new 1.94kB ± 0%: delta: 0.00%
PartialEvalHugeString-6: old 196kB ± 0%: new 196kB ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-6: old 4.30MB ± 0%: new 4.30MB ± 0%: delta: 0.00%
PartialEvalObject-6: old 19.4kB ± 0%: new 19.4kB ± 0%: delta: 0.00%
metric: allocs/op
PartialEvalComplex-6: old 3.72k ± 0%: new 3.72k ± 0%: delta: 0.00%
PartialEvalSmallString-6: old 25.0 ± 0%: new 25.0 ± 0%: delta: 0.00%
PartialEvalHugeString-6: old 38.0 ± 0%: new 38.0 ± 0%: delta: 0.00%
PartialEvalHugeInterpolatedString-6: old 25.1k ± 0%: new 25.1k ± 0%: delta: 0.00%
PartialEvalObject-6: old 179 ± 0%: new 179 ± 0%: delta: 0.00%

---

## Package: stdlib
metric: time/op
TmAllTrueLiteralList-6: old 517µs ±15%: new 495µs ±12%: delta: 0.00%
TmAllTrueFuncall-6: old 15.3µs ±16%: new 14.9µs ±11%: delta: 0.00%
TmAnyTrueLiteralList-6: old 3.70ms ± 8%: new 3.75ms ± 8%: delta: 0.00%
TmAnyTrueFuncall-6: old 14.8µs ± 7%: new 15.9µs ±11%: delta: 7.75%
TmTernary-6: old 1.75µs ±22%: new 1.95µs ±16%: delta: 11.82%
TmTryUnknownFunc-6: old 1.40µs ± 1%: new 1.56µs ± 8%: delta: 10.90%
TmTryUnknownVariable-6: old 1.50µs ±16%: new 1.52µs ± 9%: delta: 0.00%
TmTryUnknownObjectKey-6: old 1.97µs ±10%: new 1.80µs ±11%: delta: -8.58%
SlugifyLargeList-6: old 1.84ms ± 4%: new 1.73ms ± 4%: delta: -5.55%
metric: alloc/op
TmAllTrueLiteralList-6: old 320kB ± 0%: new 320kB ± 0%: delta: 0.00%
TmAllTrueFuncall-6: old 10.4kB ± 0%: new 10.4kB ± 0%: delta: 0.00%
TmAnyTrueLiteralList-6: old 2.09MB ± 0%: new 2.09MB ± 0%: delta: 0.00%
TmAnyTrueFuncall-6: old 10.5kB ± 0%: new 10.5kB ± 0%: delta: 0.00%
TmTernary-6: old 1.18kB ± 0%: new 1.18kB ± 0%: delta: 0.00%
TmTryUnknownFunc-6: old 784B ± 0%: new 784B ± 0%: delta: 0.00%
TmTryUnknownVariable-6: old 768B ± 0%: new 768B ± 0%: delta: 0.00%
TmTryUnknownObjectKey-6: old 952B ± 0%: new 952B ± 0%: delta: 0.00%
SlugifyLargeList-6: old 615kB ± 0%: new 614kB ± 0%: delta: 0.00%
metric: allocs/op
TmAllTrueLiteralList-6: old 5.93k ± 0%: new 5.93k ± 0%: delta: 0.00%
TmAllTrueFuncall-6: old 265 ± 0%: new 265 ± 0%: delta: 0.00%
TmAnyTrueLiteralList-6: old 59.6k ± 0%: new 59.6k ± 0%: delta: 0.00%
TmAnyTrueFuncall-6: old 267 ± 0%: new 267 ± 0%: delta: 0.00%
TmTernary-6: old 27.0 ± 0%: new 27.0 ± 0%: delta: 0.00%
TmTryUnknownFunc-6: old 21.0 ± 0%: new 21.0 ± 0%: delta: 0.00%
TmTryUnknownVariable-6: old 20.0 ± 0%: new 20.0 ± 0%: delta: 0.00%
TmTryUnknownObjectKey-6: old 23.0 ± 0%: new 23.0 ± 0%: delta: 0.00%
SlugifyLargeList-6: old 18.0k ± 0%: new 18.0k ± 0%: delta: 0.00%

---

## Package: tg
metric: time/op
ModuleDiscovery-6: old 48.2ms ±17%: new 53.8ms ±11%: delta: 11.67%
metric: alloc/op
ModuleDiscovery-6: old 36.7MB ± 0%: new 36.7MB ± 0%: delta: 0.00%
metric: allocs/op
ModuleDiscovery-6: old 444k ± 0%: new 445k ± 0%: delta: 0.01%

---

@mcalhoun mcalhoun force-pushed the chore/mcalhoun/speed-up-ci-jobs branch from 4798329 to 0ee2e85 Compare October 25, 2025 15:53
@github-actions
Copy link

github-actions bot commented Oct 25, 2025

Preview of ubuntu/go tests in 9e486cf

🔍 View Details on Terramate Cloud

.

@github-actions
Copy link

github-actions bot commented Oct 25, 2025

Preview of macos-ventura/go tests in 9e486cf

🔍 View Details on Terramate Cloud

.

soerenmartius
soerenmartius previously approved these changes Oct 25, 2025
Copy link
Contributor

@soerenmartius soerenmartius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@snakster snakster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work 👍 I have only one suggestion regarding benchmark discovery.

@mcalhoun mcalhoun force-pushed the feat/mcalhoun/sc-15988/new-dependency-flags branch from 9cab623 to 154ec5b Compare October 27, 2025 13:54
@mcalhoun mcalhoun force-pushed the chore/mcalhoun/speed-up-ci-jobs branch from 0ee2e85 to d97549e Compare October 27, 2025 13:54
@mcalhoun mcalhoun changed the base branch from feat/mcalhoun/sc-15988/new-dependency-flags to graphite-base/2237 October 27, 2025 17:07
@mcalhoun mcalhoun force-pushed the chore/mcalhoun/speed-up-ci-jobs branch from d97549e to b7c3076 Compare October 27, 2025 17:07
@mcalhoun mcalhoun changed the base branch from graphite-base/2237 to main October 27, 2025 17:07
@mcalhoun mcalhoun dismissed soerenmartius’s stale review October 27, 2025 17:07

The base branch was changed.

@mcalhoun mcalhoun force-pushed the chore/mcalhoun/speed-up-ci-jobs branch from b7c3076 to 8442d85 Compare October 27, 2025 17:13
cursor[bot]

This comment was marked as outdated.

@mcalhoun mcalhoun force-pushed the chore/mcalhoun/speed-up-ci-jobs branch 2 times, most recently from 804db89 to 7967643 Compare October 27, 2025 17:20
cursor[bot]

This comment was marked as outdated.

@mcalhoun mcalhoun force-pushed the chore/mcalhoun/speed-up-ci-jobs branch from 7967643 to 9e486cf Compare October 27, 2025 23:10
echo "result<<EOF" >> $GITHUB_OUTPUT
echo "$(make bench/check 'new=${{ github.event.pull_request.head.sha }}' 'old=${{ github.event.pull_request.base.ref }}')" >> $GITHUB_OUTPUT
# Sort and combine all benchmark result files
find benchmark-results -name 'benchmark-result.txt' -type f | sort | while read -r result_file; do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Suppress Error Messages for Missing Directories

If the benchmark-results directory doesn't exist (e.g., when all benchmark jobs fail or download-artifact step fails despite continue-on-error), the find command will output an error message "find: 'benchmark-results': No such file or directory" which will be captured in GITHUB_OUTPUT and displayed in the PR comment. The command should check if the directory exists first, or redirect stderr to /dev/null.

Fix in Cursor Fix in Web

Copy link
Contributor

@snakster snakster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@mcalhoun mcalhoun merged commit 770d412 into main Oct 27, 2025
17 checks passed
Copy link
Contributor Author

Merge activity

@mcalhoun mcalhoun deleted the chore/mcalhoun/speed-up-ci-jobs branch October 27, 2025 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants