[asan] Re-exec without ASLR if needed on 64-bit Linux

[thurstond]

This generalizes #131975 to non-32-bit Linux (i.e., 64-bit Linux).

This works around an edge case in 64-bit Linux, whereby the memory layout is incompatible if the stack size is unlimited AND ASLR entropy is 31+ bits (see
google/sanitizers#856 (comment)).

More generally, this "re-exec without ASLR if layout is incompatible" is a hammer that can work around most shadow mapping issues, without incurring the overhead of using a dynamic shadow.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Thurston Dang (thurstond)

Changes

This generalizes #131975 to non-32-bit Linux (i.e., 64-bit Linux).

This works around an edge case in 64-bit Linux, where the memory layout is incompatible if the stack size is unlimited AND ASLR entropy is 32 bits (see
google/sanitizers#856 (comment)).

More generally, this "re-exec if layout is incompatible" is a hammer that can work around most shadow mapping issues, without the overhead of using a dynamic shadow.

---

Full diff: https://github.com/llvm/llvm-project/pull/132682.diff

1 Files Affected:

• (modified) compiler-rt/lib/asan/asan_shadow_setup.cpp (+8-6)

diff --git a/compiler-rt/lib/asan/asan_shadow_setup.cpp b/compiler-rt/lib/asan/asan_shadow_setup.cpp
index e66b8af1d2c30..ba61dc2c7fa6e 100644
--- a/compiler-rt/lib/asan/asan_shadow_setup.cpp
+++ b/compiler-rt/lib/asan/asan_shadow_setup.cpp
@@ -109,12 +109,14 @@ void InitializeShadowMemory() {
     ProtectGap(kShadowGap2Beg, kShadowGap2End - kShadowGap2Beg + 1);
     ProtectGap(kShadowGap3Beg, kShadowGap3End - kShadowGap3Beg + 1);
   } else {
-    // The shadow mappings can shadow the entire user address space. However,
-    // on 32-bit systems, the maximum ASLR entropy (currently up to 16-bits
-    // == 256MB) is a significant chunk of the address space; reclaiming it by
-    // disabling ASLR might allow chonky binaries to run.
-    if (sizeof(uptr) == 32)
-      TryReExecWithoutASLR();
+    // ASan's mappings can usually shadow the entire address space, even with
+    // maximum ASLR entropy. However:
+    // - On 32-bit systems, the maximum ASLR entropy (currently up to 16-bits
+    //   == 256MB) is a significant chunk of the address space; reclaiming it
+    //   by disabling ASLR might allow chonky binaries to run.
+    // - On 64-bit systems, some settings (e.g., for Linux, unlimited stack
+    //   size plus maximum ASLR entropy) can lead to an incompatible layout.
+    TryReExecWithoutASLR();
 
     Report(
         "Shadow memory range interleaves with an existing memory mapping. "