Repository navigation
bench(sirun): make benchmark workloads deterministic - #9715
Conversation
Overall package sizeSelf size: 8.74 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.4.0 | 127.33 kB | 447.04 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The shard allocator preserves the current 138-variant suite across six groups, and the changed sampling and PostgreSQL workloads complete realistic high-volume runs without behavioral errors. GraphQL preflight execution could not be completed because this checkout lacks the pre-existing versions/graphql module path; metadata and syntax validation passed for that path.
📊 Validated against 15 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit a22a916 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
✅ All CI checks and tests passed. Datadog automation helped this PR pass. 🎉 All green!🧪 All tests passed 🔄 Datadog retried 1 test - 1 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: 594f68f | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-09-02 20:23:50 Comparing candidate commit 594f68f in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2308 metrics, 5 unstable metrics.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f3f37fc79
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
More details
The weighted allocator places all 137 CPU reservations into six non-overlapping shards, while delayed/aborted requests and the changed GraphQL, AppSec, IAST, WAF, and sampling paths preserve fixed-work completion and expected outputs.
📊 Validated against 10 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit f951fd4 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
More details
The deterministic workload changes behaved correctly in the executable scenarios: the shared HTTP client completed exact warmup and measured phases, WAF request and attack samples validated, and the real suite CPU weights packed into configured shards. GraphQL and debugger end-to-end runs could not be completed because this checkout lacks generated version workspaces and the debugger runtime setup.
📊 Validated against 8 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit bcf2105 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
bcf2105 to
ecf39b4
Compare
e8957e2 to
1467b79
Compare
5e80358 to
0bd1122
Compare
Wall-clock token refills, scheduler turns, GC pacing, and fixed cross-runtime counts made identical samples complete different work or spend a disproportionate share in startup, hiding regressions behind wide confidence intervals. Allocation-heavy variants use deterministic GC only where collection pacing is the noise source. Runtime-specific sizing keeps GraphQL under the unchanged startup guard without pushing slower Node versions over one minute.
Several variants run 1.5–3x beyond their target while low sample counts hide across-run drift. Rebalance total work around a 10-sample floor where variance permits, and pin exporter/GC behavior that otherwise bypasses GraphQL trace capture or destabilizes allocation-heavy runs.
Reducing both live IAST workloads by a full third lets load and setup exceed the 10% startup-share ceiling, so Sirun aborts before collecting results.
Node 24 alternates between discrete GC schedules for this allocation-heavy resolver workload, moving instructions by 3% and execution time by 12% across identical samples.
0bd1122 to
594f68f
Compare
There was a problem hiding this comment.
More details
The fixed-work loops resolve complete Node-specific operations before measurement. The GraphQL workload uses the public exporter path for traced and no-op variants.
🤖 Datadog Autotest · Commit 594f68f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
* bench(sirun): stabilize benchmark workloads Wall-clock token refills, scheduler turns, GC pacing, and fixed cross-runtime counts made identical samples complete different work or spend a disproportionate share in startup, hiding regressions behind wide confidence intervals. Allocation-heavy variants use deterministic GC only where collection pacing is the noise source. Runtime-specific sizing keeps GraphQL under the unchanged startup guard without pushing slower Node versions over one minute. * bench(sirun): rebalance benchmark runtimes and stability Several variants run 1.5–3x beyond their target while low sample counts hide across-run drift. Rebalance total work around a 10-sample floor where variance permits, and pin exporter/GC behavior that otherwise bypasses GraphQL trace capture or destabilizes allocation-heavy runs. * bench(sirun): keep IAST workloads above startup guard Reducing both live IAST workloads by a full third lets load and setup exceed the 10% startup-share ceiling, so Sirun aborts before collecting results. * bench(sirun): stabilize GraphQL collapse-off samples Node 24 alternates between discrete GC schedules for this allocation-heavy resolver workload, moving instructions by 3% and execution time by 12% across identical samples.
* bench(sirun): stabilize benchmark workloads Wall-clock token refills, scheduler turns, GC pacing, and fixed cross-runtime counts made identical samples complete different work or spend a disproportionate share in startup, hiding regressions behind wide confidence intervals. Allocation-heavy variants use deterministic GC only where collection pacing is the noise source. Runtime-specific sizing keeps GraphQL under the unchanged startup guard without pushing slower Node versions over one minute. * bench(sirun): rebalance benchmark runtimes and stability Several variants run 1.5–3x beyond their target while low sample counts hide across-run drift. Rebalance total work around a 10-sample floor where variance permits, and pin exporter/GC behavior that otherwise bypasses GraphQL trace capture or destabilizes allocation-heavy runs. * bench(sirun): keep IAST workloads above startup guard Reducing both live IAST workloads by a full third lets load and setup exceed the 10% startup-share ceiling, so Sirun aborts before collecting results. * bench(sirun): stabilize GraphQL collapse-off samples Node 24 alternates between discrete GC schedules for this allocation-heavy resolver workload, moving instructions by 3% and execution time by 12% across identical samples.
* bench(sirun): stabilize benchmark workloads Wall-clock token refills, scheduler turns, GC pacing, and fixed cross-runtime counts made identical samples complete different work or spend a disproportionate share in startup, hiding regressions behind wide confidence intervals. Allocation-heavy variants use deterministic GC only where collection pacing is the noise source. Runtime-specific sizing keeps GraphQL under the unchanged startup guard without pushing slower Node versions over one minute. * bench(sirun): rebalance benchmark runtimes and stability Several variants run 1.5–3x beyond their target while low sample counts hide across-run drift. Rebalance total work around a 10-sample floor where variance permits, and pin exporter/GC behavior that otherwise bypasses GraphQL trace capture or destabilizes allocation-heavy runs. * bench(sirun): keep IAST workloads above startup guard Reducing both live IAST workloads by a full third lets load and setup exceed the 10% startup-share ceiling, so Sirun aborts before collecting results. * bench(sirun): stabilize GraphQL collapse-off samples Node 24 alternates between discrete GC schedules for this allocation-heavy resolver workload, moving instructions by 3% and execution time by 12% across identical samples.
* bench(sirun): stabilize benchmark workloads Wall-clock token refills, scheduler turns, GC pacing, and fixed cross-runtime counts made identical samples complete different work or spend a disproportionate share in startup, hiding regressions behind wide confidence intervals. Allocation-heavy variants use deterministic GC only where collection pacing is the noise source. Runtime-specific sizing keeps GraphQL under the unchanged startup guard without pushing slower Node versions over one minute. * bench(sirun): rebalance benchmark runtimes and stability Several variants run 1.5–3x beyond their target while low sample counts hide across-run drift. Rebalance total work around a 10-sample floor where variance permits, and pin exporter/GC behavior that otherwise bypasses GraphQL trace capture or destabilizes allocation-heavy runs. * bench(sirun): keep IAST workloads above startup guard Reducing both live IAST workloads by a full third lets load and setup exceed the 10% startup-share ceiling, so Sirun aborts before collecting results. * bench(sirun): stabilize GraphQL collapse-off samples Node 24 alternates between discrete GC schedules for this allocation-heavy resolver workload, moving instructions by 3% and execution time by 12% across identical samples.
4 participants
Summary
Make non-AppSec Sirun samples complete fixed work and account for per-variant CPU reservations.
Why
Wall-clock rate limiting, async scheduler turns, ambient tracer disabling, and predictable GC made identical samples execute different work or no tracing at all. GraphQL also reached through a private processor chain absent on the no-op tracer.