Skip to content

chore: refresh benchmarks for v1.26.0 - #1002

Merged
jdx merged 1 commit into
mainfrom
bench-refresh
Jul 7, 2026
Merged

chore: refresh benchmarks for v1.26.0#1002
jdx merged 1 commit into
mainfrom
bench-refresh

Conversation

@jdx

@jdx jdx commented Jul 7, 2026

Copy link
Copy Markdown
Owner

🤖 Refreshed benchmarks

benchmarks/results.json was pinned to aube 1.25.2; the workspace is now 1.26.0. Re-ran mise run bench:bump on the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regenerated benchmarks/results.json plus the README BENCH_RATIOS block. The benchmark matrix pins aube's GVS mode via npm_config_enable_global_virtual_store=true|false (the auto-synthesized env alias for the enableGlobalVirtualStore setting), so GitHub Actions' inherited CI=true environment does not change whether aube runs with GVS enabled or disabled.

Benchmark changes

Versions:

  • aube: 1.25.2 -> 1.26.0
  • pnpm: 11.9.0 -> 11.10.0

Public ratios: warm installs vs Bun 5x -> 3x; warm installs vs pnpm 8x -> 6x; repeat test vs Bun 7x -> 3x; repeat test vs pnpm 39x -> 28x.

Benchmark aube bun deno pnpm npm yarn
Fresh install (warm cache) 319ms -> 380ms (+19%) 1491ms -> 1154ms (-23%) 1444ms -> 1353ms (-6%) 2443ms -> 2212ms (-9%) 7194ms -> 8656ms (+20%) 8491ms -> 8864ms (+4%)
Fresh install (cold cache) 14635ms -> 17450ms (+19%) 5096ms -> 7541ms (+48%) 9148ms -> 8449ms (-8%) 12995ms -> 14387ms (+11%) 10892ms -> 11424ms (+5%) 13179ms -> 15374ms (+17%)
npm install && npm run test 10ms -> 15ms (+50%) 66ms -> 48ms (-27%) 81ms -> 71ms (-12%) 388ms -> 419ms (+8%) 825ms -> 849ms (+3%) 1823ms -> 1834ms (+1%)

Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.

Once merged to main, the updated bench results flow into the next release-plz-pr run automatically.


Generated by the bench-refresh workflow.


Note

Low Risk
Docs and benchmark artifacts only; no runtime or install logic changes.

Overview
Re-runs the hermetic benchmark suite and replaces benchmarks/results.json so pinned versions match the workspace (aube 1.26.0, pnpm 11.10.0) and all three scenarios (warm/cold fresh install, repeat install && test) pick up new timings and stats.

The README BENCH_RATIOS block is regenerated from those numbers, so public speed claims shift (e.g. warm install vs pnpm 8x → 6x, vs Bun 5x → 3x; repeat test vs pnpm 39x → 28x, vs Bun 7x → 3x).

Reviewed by Cursor Bugbot for commit acc55d0. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR refreshes benchmark data in benchmarks/results.json (timestamp, pnpm version, and timing/stats for gvs-warm, gvs-cold, and install-test scenarios) and updates corresponding "Fast installs" speed multiplier claims in README.md.

Changes

Benchmark refresh

Layer / File(s) Summary
Refreshed benchmark data and README claims
benchmarks/results.json, README.md
Timestamp, pnpm version, and per-manager timing/stats for gvs-warm, gvs-cold, and install-test rows are refreshed; README warm-install and repeat-test speed multipliers are updated to match.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

  • jdx/aube#820: Another refresh of the same README benchmark ratios and benchmarks/results.json rows.
  • jdx/aube#841: Overlapping updates to the same README ratios and benchmarks/results.json timing data.
  • jdx/aube#851: Same benchmark refresh pattern touching README ratios and results.json rows.

Poem

Numbers hop and timings shift,
Warm installs get another lift,
pnpm grows a version taller,
README claims a touch smaller—err, faster!
A rabbit's work, refreshed and neat 🐇✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: refreshing benchmark data for v1.26.0.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Routine benchmark refresh bumping aube from 1.25.2 → 1.26.0 and pnpm from 11.9.0 → 11.10.0, with results.json and the README BENCH_RATIOS block regenerated from a hermetic Verdaccio run.

  • All four README speed-ratio claims (6x/3x warm, 28x/3x repeat-test vs pnpm and Bun) are arithmetically correct against the new results.json values, and every values entry matches its stats.mean rounded to milliseconds.
  • Cold-install times regressed broadly across all tools except deno (bun +48%, aube +19%, pnpm +11%, yarn +17%), which is consistent with the network-jitter caveat the PR description already calls out; these numbers are not surfaced in the README.
  • aube's warm-install rose from 319 ms to 380 ms (+19%) while all competitors improved, halving the warm-install lead — the prior review thread covers this and whether it warrants investigation before landing.

Confidence Score: 5/5

Safe to merge — this is a pure data update with no logic changes; the JSON values are internally consistent and the README ratios are arithmetically correct.

Every changed value in results.json matches its stats.mean rounded to milliseconds, the four README ratio claims check out against the new data, and no schema or structural changes were introduced. The only open question — aube's warm-install regression and whether it reflects a real v1.26.0 slowdown — is already tracked in the prior review thread.

No files require special attention beyond what the PR description already flags about cold-install network noise.

Important Files Changed

Filename Overview
benchmarks/results.json Updated benchmark data for aube 1.26.0 and pnpm 11.10.0; all values fields match their corresponding stats.mean (rounded to ms), and no structural schema changes were made
README.md BENCH_RATIOS block updated to reflect new numbers; all four stated ratios (6x/3x warm, 28x/3x repeat-test vs pnpm and Bun) are arithmetically correct given the new benchmark values

Reviews (2): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile

Comment thread benchmarks/results.json Outdated
Comment on lines +23 to +35
"key": "gvs-warm",
"label": "Fresh install (warm cache)",
"values": {
"aube": 319,
"bun": 1491,
"deno": 1444,
"pnpm": 2443,
"npm": 7194,
"yarn": 8491
"aube": 552,
"bun": 1481,
"deno": 1387,
"pnpm": 2238,
"npm": 7428,
"yarn": 8514
},
"stats": {
"aube": {
"text": "0.319s ± 0.076s",
"mean": 0.31892467918000006,
"stddev": 0.07556120161164992,
"min": 0.23709043908000002,
"max": 0.45854669708000007
"text": "0.552s ± 0.093s",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Significant warm-install regression warrants investigation before landing

aube's warm-cache fresh-install time jumped from 319 ms to 552 ms (+73%) between v1.25.2 and v1.26.0. All competing tools stayed flat or improved in the same run, so the hardware baseline is clean. This halved the advertised warm-install advantage (8x→4x vs pnpm, 5x→3x vs Bun). If v1.26.0 intentionally traded warm-install speed for another property (e.g. correctness, additional resolve passes, or a new default behavior enabled), this should be noted in the release notes. Otherwise, it may be worth checking whether a regression was introduced in the frozen fast-path (install/frozen.rs) or the linker's symlink pass before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code

@jdx
jdx enabled auto-merge (squash) July 7, 2026 13:26
@jdx
jdx force-pushed the bench-refresh branch from 25ad7e6 to acc55d0 Compare July 7, 2026 16:22
@jdx
jdx merged commit a366306 into main Jul 7, 2026
18 checks passed
@jdx
jdx deleted the bench-refresh branch July 7, 2026 16:29
@cursor cursor Bot mentioned this pull request Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant