chore: refresh benchmarks for v1.26.0 - #1002
Conversation
📝 WalkthroughWalkthroughThis PR refreshes benchmark data in benchmarks/results.json (timestamp, pnpm version, and timing/stats for gvs-warm, gvs-cold, and install-test scenarios) and updates corresponding "Fast installs" speed multiplier claims in README.md. ChangesBenchmark refresh
Estimated code review effort: 1 (Trivial) | ~3 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryRoutine benchmark refresh bumping aube from 1.25.2 → 1.26.0 and pnpm from 11.9.0 → 11.10.0, with
Confidence Score: 5/5Safe to merge — this is a pure data update with no logic changes; the JSON values are internally consistent and the README ratios are arithmetically correct. Every changed value in results.json matches its stats.mean rounded to milliseconds, the four README ratio claims check out against the new data, and no schema or structural changes were introduced. The only open question — aube's warm-install regression and whether it reflects a real v1.26.0 slowdown — is already tracked in the prior review thread. No files require special attention beyond what the PR description already flags about cold-install network noise. Important Files Changed
Reviews (2): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile |
| "key": "gvs-warm", | ||
| "label": "Fresh install (warm cache)", | ||
| "values": { | ||
| "aube": 319, | ||
| "bun": 1491, | ||
| "deno": 1444, | ||
| "pnpm": 2443, | ||
| "npm": 7194, | ||
| "yarn": 8491 | ||
| "aube": 552, | ||
| "bun": 1481, | ||
| "deno": 1387, | ||
| "pnpm": 2238, | ||
| "npm": 7428, | ||
| "yarn": 8514 | ||
| }, | ||
| "stats": { | ||
| "aube": { | ||
| "text": "0.319s ± 0.076s", | ||
| "mean": 0.31892467918000006, | ||
| "stddev": 0.07556120161164992, | ||
| "min": 0.23709043908000002, | ||
| "max": 0.45854669708000007 | ||
| "text": "0.552s ± 0.093s", |
There was a problem hiding this comment.
Significant warm-install regression warrants investigation before landing
aube's warm-cache fresh-install time jumped from 319 ms to 552 ms (+73%) between v1.25.2 and v1.26.0. All competing tools stayed flat or improved in the same run, so the hardware baseline is clean. This halved the advertised warm-install advantage (8x→4x vs pnpm, 5x→3x vs Bun). If v1.26.0 intentionally traded warm-install speed for another property (e.g. correctness, additional resolve passes, or a new default behavior enabled), this should be noted in the release notes. Otherwise, it may be worth checking whether a regression was introduced in the frozen fast-path (install/frozen.rs) or the linker's symlink pass before merging.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
🤖 Refreshed benchmarks
benchmarks/results.jsonwas pinned to aube1.25.2; the workspace is now1.26.0. Re-ranmise run bench:bumpon the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regeneratedbenchmarks/results.jsonplus the READMEBENCH_RATIOSblock. The benchmark matrix pins aube's GVS mode vianpm_config_enable_global_virtual_store=true|false(the auto-synthesized env alias for theenableGlobalVirtualStoresetting), so GitHub Actions' inheritedCI=trueenvironment does not change whether aube runs with GVS enabled or disabled.Benchmark changes
Versions:
Public ratios: warm installs vs Bun 5x -> 3x; warm installs vs pnpm 8x -> 6x; repeat test vs Bun 7x -> 3x; repeat test vs pnpm 39x -> 28x.
Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.
Once merged to main, the updated bench results flow into the next
release-plz-prrun automatically.Generated by the
bench-refreshworkflow.Note
Low Risk
Docs and benchmark artifacts only; no runtime or install logic changes.
Overview
Re-runs the hermetic benchmark suite and replaces
benchmarks/results.jsonso pinned versions match the workspace (aube 1.26.0, pnpm 11.10.0) and all three scenarios (warm/cold fresh install, repeatinstall && test) pick up new timings and stats.The README
BENCH_RATIOSblock is regenerated from those numbers, so public speed claims shift (e.g. warm install vs pnpm 8x → 6x, vs Bun 5x → 3x; repeat test vs pnpm 39x → 28x, vs Bun 7x → 3x).Reviewed by Cursor Bugbot for commit acc55d0. Bugbot is set up for automated code reviews on this repo. Configure here.