Portable builds in CI

[cataphract]

Description

Every Linux artifact used to be built twice: once against glibc on the old CentOS 7 images, and once against musl on Alpine. That doubled the compile jobs for each component and PHP version, and kept an ageing toolchain alive only to get an old enough glibc.

This PR builds each extension, the sidecar library and the loader once per architecture, against musl, and links them in a way that loads in both glibc and musl processes. The build scripts refuse any output that picks up a glibc symbol version. The builds now run in the parent pipeline, so the test, appsec, profiler and packaging child pipelines all use the binaries that actually ship, instead of rebuilding their own.

A few runtime fixes came with it, because the musl build changed assumptions the code had been relying on implicitly:

• The backtrace functions are looked up at runtime. datadog.log_backtrace now warns when they're unavailable instead of being silently ignored.
• A small trampoline sets PHP's pinned VM registers before ddtrace calls a VM handler directly.
• The Rust-side session-id buffers are aligned the way C expects.

The CentOS 7 images are no longer needed and are removed.

With one binary per architecture, the release bundles no longer carry a libc in their name. That is the one change users may notice:

Release asset names changed

Linux builds are now portable across glibc and musl, so the separate -linux-gnu and -linux-musl release assets are replaced by one archive per architecture:

• dd-library-php-<version>-<arch>-linux-gnu.tar.gz and …-linux-musl.tar.gz → dd-library-php-<version>-<arch>-linux.tar.gz
• …-linux-gnu-<API>[-zts|-debug].tar.gz and …-linux-musl-<API>[-zts].tar.gz → …-linux-<API>[-zts|-debug].tar.gz

datadog-setup.php already uses the new names. Scripts that download these archives directly must be updated.

Summary of job changes

Reviewer checklist

• Test coverage seems ok.
• Appropriate labels assigned.

[datadog-prod-us1-5]

 

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 8 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | benchmarks-tracer — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-php | merge-gate

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-php | publish docker image for system tests

View more details · View in GitLab

View all 8 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 68.26% (+0.00%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b0ebefa | Docs | View more details | Give us feedback!}

[pr-commenter]

Benchmarks [ tracer ]

Benchmark execution time: 2026-09-30 10:47:14

Comparing candidate commit 31a3cfb in PR branch glopes/portable-builds with baseline commit d98de1e in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 194 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-29T15:27:52.685760Z	a38379b	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-29T15:27:35.496227Z	a38379b	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[realFlowControl]

I'm skeptical about this PR for two reasons

First, I’m not convinced our current test coverage validates the build change. Two examples from a spot check:

• The main profiler PHPT suite still builds native profiler binaries separately on Alpine and Debian, rather than testing the portable musl-built profiler on glibc.
• The green benchmark result comes from tracer microbenchmarks that rebuild their own binaries rather than using the portable release artifacts.

These two checks therefore do not validate the portability and performance of the binaries we would ship. I know other jobs consume the portable artifacts.

Second (and that is my biggest concernt), this moves libc compatibility assumptions from the standard toolchain into a compatibility layer we need to maintain and validate.

For exmaple on ARM64, pthread_attr_t is 56 bytes in musl but 64 bytes in glibc. We patch the C headers, but I don’t see a fix for Rust’s libc definition. Without that fix, Rust allocates a 56-byte object, while glibc’s pthread_attr_init() writes 64 bytes into it, 8 bytes past the end. So far it looks like we are lucky, because those extra 8 bytes are in stack slots not used during that call, but that is pure luck and not a guarantee.

I would prefer to keep relying on Rust’s supported target/libc pairing rather than take on the ongoing responsibility of validating the standard library and our dependencies against a different libc.

[cataphract]

@realFlowControl I believe I've addresses most of your point, though the more abstract one, ofc remains. Please see also https://chonk.us1.prod.dog/v2/get/chonk-uploads-prod/gustavo.lopes/9d9b5f3e-4b01-4d60-884e-19bb327fffc6 for the summary of the job changes

[pr-commenter]

Benchmarks [ profiler ]

Benchmark execution time: 2026-10-01 15:12:41

Comparing candidate commit b0ebefa in PR branch glopes/portable-builds with baseline commit 457aa53 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 1 performance regressions! Performance is the same for 25 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:walk_stack/1

• 🟥 wall_time [+276.193ns; +279.495ns] or [+2.262%; +2.289%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:php-profiler-exceptions-control

• unstable cpu_system_time [-0.024ms; +8.222ms] or [-0.057%; +19.276%]
• unstable cpu_user_time [-8.097ms; +0.304ms] or [-13.079%; +0.491%]

scenario:php-profiler-exceptions-with-profiler

• unstable cpu_system_time [-7.775ms; +2.934ms] or [-16.259%; +6.136%]
• unstable cpu_usage_percentage [+19.753%; +57.843%]
• unstable cpu_user_time [-5242.899µs; +7168.899µs] or [-6.888%; +9.419%]
• unstable execution_time [-119.698ms; -42.606ms] or [-60.687%; -21.601%]

scenario:php-profiler-exceptions-with-profiler-and-timeline

• unstable cpu_system_time [-8.409ms; +1.122ms] or [-16.746%; +2.235%]
• unstable cpu_user_time [-0.092ms; +9.061ms] or [-0.124%; +12.191%]

scenario:php-profiler-timeline-memory-control

• unstable cpu_system_time [-4341.816µs; +3175.016µs] or [-9.788%; +7.157%]

scenario:php-profiler-timeline-memory-with-profiler

• unstable cpu_system_time [-29.156ms; +20.331ms] or [-6.856%; +4.781%]