Add performance conscious hardening flags to linux targets

[jjhelmus]

Build CPython and dependencies with a set of hardening flags that take into account performance.

• Enable stack protection and immediate binding for all linux builds.
• Enable source fortification for optimized glibc targets.
• Enable stack-clash protection on aarch64/x86-64 linux targets.
• Validate these hardening properties in distributions using the validate-distribution command.

Control-flow and branch protection flags are not included.

On x86-64 -fcf-protection requires modern kernels, glibc and hardware (CET). Additionally, native extensions can fail to load when they are built without CET (which most are not) if the interpreter includes these protections.

On aarch64, -mbranch-protection is of limited use unless all objects include the necessary markers. This includes the CRT and compiler runtimes which come from the base image which pre-dates these features.

The -fstack-protector flag is used rather than -fstack-protector-strong which would instrument additional functions. The later was found to have a negative effect on performance (~2-3%). Debian currently uses this flag when building CPython in the 3.13 release for Trixie.

Executable are not validate or required to be position independent (PIE). Testing found that requiring this has significant negative effects on performance (~5%). Debian builds CPython without PIE in the 3.13 release for Trixie.

closes #837

[jjhelmus]

This improves the hardening check results:

PBS 20260623 on aarch64 Linux

# curl -SLO https://github.com/astral-sh/python-build-standalone/releases/download/20260623/cpython-3.14.6+20260623-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz
# tar xf cpython-3.14.6+20260623-aarch64-unknown-linux-gnu-install_only_stripped.tar.gz
# hardening-check python/bin/python
python/bin/python:
 Position Independent Executable: yes
 Stack protected: no, not found!
 Fortify Source functions: no, only unprotected functions found!
 Read-only relocations: yes
 Immediate binding: no, not found!
 Stack clash protection: unknown, no -fstack-clash-protection instructions found
 Control flow integrity: no, not found!
 Branch Protection: no, not found!

With these changes:

# tar xf cpython-3.14.6-aarch64-unknown-linux-gnu-pgo+lto-20260625T1336.tar.zst
# hardening-check python/install/bin/python
python/install/bin/python:
Position Independent Executable: yes
Stack protected: yes
Fortify Source functions: yes (some protected functions found)
Read-only relocations: yes
Immediate binding: yes
Stack clash protection: unknown, no -fstack-clash-protection instructions found
Control flow integrity: no, not found!
Branch Protection: no, not found!

Stack clash protection is enabled but the binary lacks allocation that would exercise this protection so in cannot be detected.
Control flow integrity and branch protection and intentionally not enabled as discussed above.

[jjhelmus]

Validation is failing with:

Error: errors found
  error: python/install/bin/python3.14 is not position-independent (ELF type is ET_EXEC)
Error: Process completed with exit code 1.

This is likely a false positive as the binary is PIE rather than PIC.

[jjhelmus]

Before merging this I was to examine the performance difference these flags introduce. I will not be able to get to this until next week at the earliest.

[jjhelmus]

I benchmarked these changes using fastmark.

Using the 20260623 release of CPython 3.14.6 for x86-64 Linux with pgo+lto fastmarks on my Intel i9-9900k gave a value of 13951 (average of five runs). On the same setup after applying these flags gave a value of 13590. This is ~97.4% of the original value, so the hardening flags cause a 2-3% drop in performance. This is larger than I expected but not way off.

[indygreg]

Is there any impact on startup time?

[jjhelmus]

Is there any impact on startup time?

Not by a significant amount, <2%, and within the measured error.

Reference (20260623 release)

❯ hyperfine ./ref_python/install/bin/python -c 'pass'
Benchmark 1: ./ref_python/install/bin/python
  Time (mean ± σ):      12.4 ms ±   0.7 ms    [User: 9.6 ms, System: 2.8 ms]
  Range (min … max):    11.8 ms …  14.3 ms    231 runs

With these flags:

❯ hyperfine ./python/install/bin/python -c 'pass'
Benchmark 1: ./python/install/bin/python
  Time (mean ± σ):      12.6 ms ±   0.5 ms    [User: 9.8 ms, System: 2.7 ms]
  Range (min … max):    11.8 ms …  14.8 ms    222 runs

[jjhelmus]

Dug a little deeper into performance and -fstack-protector-strong is the primary cause of the change. As this is one of the more important hardening options I think it should be applied even with the decrease in performance.

[jjhelmus]

This is likely a false positive as the binary is PIE rather than PIC.

This is not a false positive, the x86-64 binary not position independent. Need to figure out why not.

[jjhelmus]

When BOLT is enabled CPython explicitly turns off PIE for performance

  # These flags are required to get good performance from bolt:
  CFLAGS_NODIST="$CFLAGS_NODIST -fno-pie"
  # We want to add these no-pie flags to linking executables but not shared libraries:
  LINKCC="$LINKCC -fno-pie -no-pie"

[zanieb]

uh... that's interesting. I wonder if that's still true?

[jjhelmus]

uh... that's interesting. I wonder if that's still true?

I suspect not. I'm testing the effects now.

[jjhelmus]

Expanded the test matrix here to capture any issues on other architectures+python versions. Likely a bit more that needed but I would rather catch issues here than post merge.

[jjhelmus]

Making the BOLT optimized binary position independent has a significant effect on performance:

Fastmark scores:

• 20260623 (reference): 13,951
• Hardening non-PIE : 13,590 (-2.6%)
• Hardening PIE : 12,910 (-7.5%, -4.9% from just PIE)

Startup time is also effected going to ~13.5 ms (+8.9%)

[zanieb]

PIE with BOLT is faster?

[jjhelmus]

For fastmark larger scores are better (more performant). So the best performance is without hardening flags and without PIE.

[jjhelmus]

It is worth noting that the system provided by Debian Trixie is non-PIE and uses the weaker -fstack-protector option. There is also a pythonX.Y-nopie package that explicitly ship a non-PIE interpreter but this seems unnecessary given that the default is package is also non-PIE.

[zanieb]

So now we're talking about an 8% performance regression instead? That's feeling pretty questionable.

[jjhelmus]

So now we're talking about an 8% performance regression instead? That's feeling pretty questionable.

Yes, enabling PIE and -fstack-protection-strong causes a ~8% performance regression.

A set of performance conscious hardening flags that adjusting the existing flags to leave off PIE and use the less aggressive -fstack-protection (not the strong variant) has little effect on performance, <1%, from my limited testing. I am going to get that set passing and check performance in more detail.

[jjhelmus]

The ppc64le validation failure is caused by the wrong startup file being selected (crt1.o) and linked when building a PIE executable resulting in text relocations, see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81170

[jjhelmus]

I think this set of flags is a good balance between security and performance.

With these flags the performance measured by fastmark between the 20260623 and a x86-64 Python 3.14 interpreter build with these flags is within the spread of multiple runs of the benchmark.

I'm running the full pyperformance benchmark suite overnight to get more details.

[jjhelmus]

The results from running pyperformance show little difference between the 20260623 release and 3.14 built on x86-64 with these flags. Overall the geometric mean shows building with these flags is 1.01x times faster but the 1% difference is well within the error of these benchmarks, especially as turbo is not disable on the machine I ran these on.

Startup is slightly decreased with these flags, but the python_startup_no_site benchmarks has no significant difference.

python_startup            | 12.5 ms  | 13.0 ms: 1.04x slower

[EliteTK]

Looks good generally, only some minor comments!

[EliteTK]

We don't seem to include heuristics for stack protector or fortify (could look for __stack_chk_fail and __<func>_chk I think). Maybe something to follow up on?

[jjhelmus]

Created #1172 for this follow up.