perf: Run all benchmarks for shorter time

[bantonsson]

Changes

Shortens the warmup time to 1 second and measurement time to 2 seconds for all benchmarks, and prepares them to be run in one go from the workspace root. Most benchmarks have millions or billions of iterations with a 5 second measurement, which seems a bit excessive.

Merge requirement checklist

• CONTRIBUTING guidelines followed
• Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.8%. Comparing base (b2de6cc) to head (ce2b5a8).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #2870     +/-   ##
=======================================
- Coverage   80.9%   80.8%   -0.1%     
=======================================
  Files        124     124             
  Lines      23833   23833             
=======================================
- Hits       19281   19280      -1     
- Misses      4552    4553      +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[scottgerring]

Ooh! I'd love for this to be quick enough we can run it more often. Let's see how this run goes.

[scottgerring]

Great to combine this all into one

[bantonsson]

Since this doesn't work in one go, I'm going to split it into two PRs.

[bantonsson]

Also, since the base benchmarks are being run on main they will warm up and measure for 3 + 5 seconds, so the time gains won't be as pronounced.

[bantonsson]

I just noticed that the github workflow is not switching back to the PR branch after the first invocation of boa-dev/criterion-compare-action@v3, so the two last measurements are not comparing the right things. They are comparing main to main 😱.

[scottgerring]

I just noticed that the github workflow is not switching back to the PR branch after the first invocation of boa-dev/criterion-compare-action@v3, so the two last measurements are not comparing the right things. They are comparing main to main 😱.

Yuck. Good catch. Fortunately we've only ever discussed changes from the first run - all the context stuff happens before the rest!

[bantonsson]

This now runs the benchmarks in ~50 minutes instead of ~1:05. And that is with only the changed side running faster while main still warms up for 3 seconds and measures for 5 seconds. I expect that there will be another ~15 minutes decrease when this is merged to main.

[scottgerring]

This is just what we need, and all looks good to me. Thanks for the effort! Excited to see how fast it'll be once we have it in main too.