[BOLT][AArch64] Runtime crash with `-split-functions -lite=1`

[paschalis-mpeis]

Provide a reproducer for a bug that is discussed at:

• [AArch64][BOLT] Ensure tentative code layout for cold BBs runs. #96609

Bug summary:

When -split-functions -lite=1 is used on AArch64 with the below MongoDB setup, it crashes at runtime. Doing a tentative code-layout size estimation for cold blocks does not trigger this bug (using the now-merged patch of #96609). However, the bug should not have occurred regardless. Point 2 below has more details (see here).

---

Reproducer:

Summary of reproducing options:

Reproducing option	Follow from step
Reproduce the crash directly. Uses pre-generated: • input binary (mongod.tar.gz) • bolted profile (perf.boltdata)	Step (3)
Generate sampled profile and then reproduce the crash. Uses pre-generated: • input binary (mongod.tar.gz)	Step (2)
Generate all artifacts from sources. Compiles stage-0 MongoDB and follows all steps.	Step (1)

NOTE: Find all attachments at the end of this post. The mongod binary (in mongod.tar.gz) has stripped debug data due to upload size requirements. If those are needed please follow all steps to get a binary with debug info.

1. Generate MongoDB stage-0 binary from sources:

1.A. Instructions with docker:

Get this Dockerfile for compiling mongodb on AArch64 using:

# Compile mongodb v7.0.5 with clang 18.1.8 (stage-0):
docker build --progress=plain --tag 'tmp-mdb-stage-0' .

# Then, 'pull' the binary from container to the host, given host is Ubuntu 22.04/24.04 (preferred)
docker run --rm --entrypoint cat tmp-mdb-stage-0 mongo/build/install/bin/mongod > mongod
chmod +x mongod

# once binary is retrieved the image could be deleted
docker rmi tmp-mdb-stage-0

Dockerscript tested on AArch64 Ubuntu LTS hosts:

• Noble: 24.04.1, 6.8.0-1016-aws, docker 27.3.1
• Jammy: 22.04.4 / 22.04.5, on 6.8.0-1016-aws / 6.8.0-1017-aws (metal or not), docker 24.0.5 / 25.0.3

Note: it may run on other AArch64 hosts (e.g., AL2023) but is not recommended. Prefer building on Ubuntu 22/24. After building, extract it from Docker to perform steps 2 and 3 directly on the host. Otherwise, more might be needed for generating a perf profile and continuing with the steps within the docker container.

1.B. Sample Instructions without docker (alternative):

Show details

**Note: If compiling from sources you may need to consult MongoDB documentation and/or adjust commands/patch below slightly, depending on your setup.**

Install required software and compile mongodb from sources.

The patch mongod.patch needs to be applied. It deals with a compilation error on a library dependency.

Configuration this was tested:

Software	Version
OS	Ubuntu 22.04.3 LTS
Kernel	6.8.0-1016-aws
clang	18.1.8 (++20240615103753+3b5b5c1ec4a3-1exp120240615223858.136)
Python	3.10.12

# 1. pre-requisites
sudo apt-get install build-essential
  
# if ran into errrors, you may also need:
sudo apt install \
    libcurl4-openssl-dev \
    liblzma-dev \
    libssl-dev \
    python3-pip \
    python3-venv \
    openjdk-21-jdk
  
# 2. get mongo sources  
git clone --branch r7.0.5 https://github.com/mongodb/mongo.git
  
# 3. setup python environment
python3 -m venv mongo-db-venv --prompt mongo
source mongo-db-venv/bin/activate
  
# 4. install mongod dependencies and apply needed patch  
cd mongo
pip install --upgrade pip
pip install -r etc/pip/compile-requirements.txt
git am mongo.patch # apply patch
  
# 5. compile stage-0 binary
python3 buildscripts/scons.py install-mongod \
  CC=clang-18 CXX=clang++-18 \
  CCFLAGS="-fno-omit-frame-pointer -Wno-deprecated-non-prototype -Wno-enum-constexpr-conversion" \
  LINKFLAGS="-Wl,-q"
  
cd .. # back to parent dir

---

2. Generate profile & BOLT optimize

2.a. Get YCSB harness

This is needed for initializing the database with some data, and then running a simple workload to trigger the crash.

# 1. Get maven
wget https://dlcdn.apache.org/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz
  
mkdir -p maven
tar -xf apache-maven-*-bin.tar.gz -C maven
  
# 2. Get and build YSCB from sources:  
git clone --branch production https://github.com/mongodb-labs/YCSB.git
orig_dir=$(pwd)
cd YCSB/ycsb-mongodb  
$orig_dir/maven/apache-maven-3.9.6/bin/mvn clean package

2.b. Build a dataset

# 1. Start the db service (in one tab/pane/window, as this is blocking)
mkdir -p mongo/dataset # empty dataset
./mongod --dbpath=./mongo/datase 
  
# 2. Create a dataset (in a second tab)  
cd YCSB/ycsb-mongodb
bin/ycsb \
    load mongodb -P workloads/workloada \
    -s \
    -p recordcount=20000 \
    -p operationcount=10000
  
# 3. Stop the db service (control-d)

2.c. Record a perf profile

# 1. Start db service (in one tab, as this is blocking)
./mongod --dbpath=./mongo/dataset
  
# 2. Find its process id and verify manually that you got the relevant pid:
# (sample command)
PID=$(ps uax | grep mongod | grep -v grep | awk '{print $2 }')
  
# 3. Record samples (in another tab)  
perf record -e cycles:uP -p $PID
  
# 4. Start benchmarking (in another tab)  
bin/ycsb run mongodb -P workloads/workloada \
                -s \
                -p recordcount=20000 \
                -p operationcount=10000
  
# 5. Stop perf record once YCSB is done. Also kill the db service (control-d).

---

3. Generate BOLT'ed binary

3.a. Compile LLVM sources:

Checkout latest main without the patch of #96609:

# 0. clone llvm/llvm-project and set the below var:
llvm=/path/to/llvm/dir
  
# 1. checkout the latest patch w/o the pr improvement:
git checkout cb9bacf57d5c~1
  
# 2. compile BOLT from sources (adjust commands):  
cmake -G Ninja .. \
        -DLLVM_TARGETS_TO_BUILD=AArch64 \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_ASSERTIONS=ON \
        -DLLVM_ENABLE_PROJECTS='clang;bolt;lld' \
        -DLLVM_USE_LINKER=lld
ninja ...

3.b. Generate BOLT'ed binary

# 1. convert profile:  
$llvm/build/bin/perf2bolt -p perf.data -o perf.boltdata --nl mongod
  
# 2. compile bolted binary:  
$llvm/build/bin/llvm-bolt mongod -o mongod.bolt --data perf.boltdata -lite=1 -split-functions

3.c. Observe runtime crash..

# 1. start the database service (in one tab, as the below is blocking)  
./mongod.bolt --dbpath=./mongo/dataset  
  
# 2. start benchmarking  (in another tab)
cd YCSB/ycsb-mongodb
./bin/ycsb run mongodb -P workloads/workloada \
                -s -threads 1 \
                -p recordcount=20000 \
                -p operationcount=10000

---

Binaries and aux files:

All files are hosted on this Github Gist: 9eb878f73e18fb9d3f996ae7c59d4792.

Binary	Description	Details
mongod.tar.gz	Stage-0 binary of mongod service.	Extract with tar -xvf mongod.tar.gz
perf.boltdata	Converted fdata BOLT profile.	Needed for optimizing mongod binary with BOLT.
mongod.patch	Patch for mongodb sources.	Needed for compiling mongodb with clang-18.
llvm-bolt.txt	Output of llvm-bolt.	Not needed for compilation/optimization.
perf2bolt.txt	Output of perf2bolt profile conversion.	Not needed for compilation/optimization.

Binary	Checksum	Details
mongod	f379208ed066b85dfb30d317e1758e9e	Generate using md5sum mongod

[llvmbot]

@llvm/issue-subscribers-bolt

Author: Paschalis Mpeis (paschalis-mpeis)

## Bug summary: This issue provides a reproducer for a bug that is discussed at: - #96609

When -split-functions -lite=1 is used on AArch64 and the below MongoDB setup, it crashes at runtime. Doing a tentative code-layout size estimation hides this issue probably in most cases (the patch provided by #96609). However, the bug should not have occurred without that improvement.
See point (2) here for more details.

---

Reproducing Instructions:

Instruction summary:

• To reproduce the crash directly, follow instructions from step (3). You must supply the input binary (mongod.tar.gz) and the bolt profile (perf.boltdata) to bolt-optimize the binary and finally use it to run a workload.

• To generate the stage-0 binary, follow instructions from step (1).

• To generate the sampled profile follow instructions from step (2). You must supply the input binary (mongod.tar.gz).

NOTE: Find all attachments at the end of this post. The mongod binary (in mongod.tar.gz) has stripped debug data due to upload size requirements. If those are needed please follow all steps to get a binary with debug info.

1. Generate stage-0 binary from sources:

Install required software and compile mongodb from sources.

The patch mongod.patch needs to be applied. It deals with a compilation error on a library dependency.

# 1. pre-requisites
sudo apt-get install build-essential
  
# if ran into errrors, you may also need:
sudo apt install \
    libcurl4-openssl-dev \
    liblzma-dev \
    libssl-dev \
    python3-pip \
    python3-venv \
    openjdk-21-jdk
  
# 2. get mongo sources  
git clone --branch r7.0.5 https://github.com/mongodb/mongo.git
  
# 3. setup python environment
python3 -m venv mongo-db-venv --prompt mongo
source mongo-db-venv/bin/activate
  
# 4. install mongod dependencies and apply needed patch  
cd mongo
pip install --upgrade pip
pip install -r etc/pip/compile-requirements.txt
git am mongo.patch # apply patch
  
# 5. compile stage-0 binary
python3 buildscripts/scons.py install-mongod \
  CC=clang-18 CXX=clang++-18 \
  CCFLAGS="-fno-omit-frame-pointer -Wno-deprecated-non-prototype -Wno-enum-constexpr-conversion" \
  LINKFLAGS="-Wl,-q"
  
cd .. # back to parent dir

2. Generate profile and optimize with BOLT

2.a. Get YCSB harness

Needed to load the database with some dummy data, and then run a simple workload that will demonstrate the crash.

# 1. get maven
wget https://dlcdn.apache.org/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz
  
mkdir -p maven
tar -xf apache-maven-*-bin.tar.gz -C maven
  
# 2. Get and build YSCB from sources:  
git clone --branch production https://github.com/mongodb-labs/YCSB.git
orig_dir=$(pwd)
cd YCSB/ycsb-mongodb  
$orig_dir/maven/apache-maven-3.9.6/bin/mvn clean package

2.b. Build a dataset

# 1. start the database service (in one tab)
mkdir mongo/dataset # empty dataset
./mongo/build/install/bin/mongod --dbpath=./mongo/datase 
  
# 2. load dataset (in a second tab)  
cd YCSB/ycsb-mongodb
bin/ycsb \
    load mongodb -P workloads/workloada \
    -s \
    -p recordcount=20000 \
    -p operationcount=10000
  
# 3. kill the database service (control-d)

2.c. Record a perf profile

# 1. start the database service (in one tab)
mkdir mongo/dataset # empty dataset
./mongo/build/install/bin/mongod --dbpath=./mongo/dataset
  
# 2. find it's process id (VERIFY with ps):  
# (sample command)  
PID=$(ps uax | grep /build/install/bin/mongod | grep -v grep | awk '{print $2 }')
  
# 3. start recording samples (in another tab)  
perf record -e cycles:uP -p $PID
  
# 4. start benchmarking (in another tab)  
bin/ycsb run mongodb -P workloads/workloada \
                -s \
                -p recordcount=20000 \
                -p operationcount=10000
  
# 5. stop perf record once YCSB is done, and  
#    also kill the database service.

3. Generate BOLT'ed binary

3.a. Compile LLVM sources:

Checkout latest main or if checking out the pull request #96609 ensure you don't include the fixes.

# assume you cloned llvm/llvm-project at:  
llvm=PATH_TO_LLVM_SOURCES
  
# 1. checkout latest main. otherwise, checkout PR96609, and go  
# back 2 commits so you don't include the fix.  
gh pr checkout 96609
git checkout HEAD~2
  
# 2. compile BOLT from sources (adjust commands):  
cmake -G Ninja \
        ... \
        -DLLVM_TARGETS_TO_BUILD=AArch64 \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLVM_ENABLE_ASSERTIONS=ON \
        -DLLVM_ENABLE_PROJECTS='clang;bolt;lld' \
        -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
        -DLLVM_USE_LINKER=lld
  
ninja ...

3.b. Generate BOLT'ed binary

# 1. convert profile:  
$llvm/build/bin/perf2bolt -p perf.data -o perf.boltdata --nl mongo/build/install/bin/mongod
  
# 2. compile bolted binary:  
$llvm/build/bin/llvm-bolt mongo/build/install/bin/mongod -o mongo/build/install/bin/mongod.bolt --data perf.boltdata -lite=1 -split-functions

3.c. Observe runtime crash..

# 1. start the database service (in one tab)  
./mongo/build/install/bin/mongod.bolt --dbpath=./mongo/dataset  
  
# 2. start benchmarking  
cd YCSB/ycsb-mongodb
./bin/ycsb run mongodb -P workloads/workloada \
                -s -threads 1 \
                -p recordcount=20000 \
                -p operationcount=10000

---

Binaries and aux files:

All files are hosted on this Github Gist: 9eb878f73e18fb9d3f996ae7c59d4792.

Binary	Description	Details
mongod.tar.gz	Stage-0 binary of mongod service.	Extract with tar -xvf mongod.tar.gz
perf.boltdata	Converted fdata BOLT profile.	Needed for optimizing mongod binary with BOLT.
mongod.patch	Patch for mongodb sources.	Needed for compiling mongodb with clang-18.
llvm-bolt.txt	Output of llvm-bolt.	Not needed for compilation/optimization.
perf2bolt.txt	Output of perf2bolt profile conversion.	Not needed for compilation/optimization.

Binary	Checksum
mongodb	f379208ed066b85dfb30d317e1758e9e

[yota9]

Hello! Honestly I'm not ready to run the binary to observe crash, but if you give me some information on where crash occured e.g. using GDB I might check it and maybe help you to find the root case of the problem

[peterwaller-arm]

Here is an example, unfortunately it's a PIE binary so ASLR is in effect, I also cannot be certain that this sample crash comes from the linked binary in the issue body, in case you are inspecting that one.

Thread 1 "mongod" received signal SIGSEGV, Segmentation fault.
0x0000aaaab2a60860 in mongo::LockManager::lock(mongo::ResourceId, mongo::LockRequest*, mongo::LockMode) ()
(gdb) bt
h1.0  0x0000aaaab2a60860 in mongo::LockManager::lock(mongo::ResourceId, mongo::LockRequest*, mongo::LockMode) ()
h1.1  0x0000aaaab2a602a4 in mongo::LockerImpl::_lockBegin(mongo::OperationContext*, mongo::ResourceId, mongo::LockMode) ()
h1.2  0x0000aaaab2a5ff54 in mongo::LockerImpl::lock(mongo::OperationContext*, mongo::ResourceId, mongo::LockMode, mongo::Date_t) ()
h1.3  0x0000aaaab2a54500 in void std::vector[mongo::CollectionNamespaceOrUUIDLock, std::allocator<mongo::CollectionNamespaceOrUUIDLock] >::_M_realloc_insert[mongo::OperationContext*&, mongo::NamespaceStringOrUUID const&, mongo::LockMode&, mongo::Date_t&](*gnu_cxx::*normal_iterator[mongo::CollectionNamespaceOrUUIDLock*, std::vector<mongo::CollectionNamespaceOrUUIDLock, std::allocator<mongo::CollectionNamespaceOrUUIDLock] > >, mongo::OperationContext*&, mongo::NamespaceStringOrUUID const&, mongo::LockMode&, mongo::Date_t&) ()
h1.4  0x0000aaaab2a5389c in mongo::AutoGetCollection::AutoGetCollection(mongo::OperationContext*, mongo::NamespaceStringOrUUID const&, mongo::LockMode, mongo::auto_get_collection::OptionsWithSecondaryCollections, bool) ()
h1.5  0x0000aaaab2a51f74 in mongo::AutoGetCollection::AutoGetCollection(mongo::OperationContext*, mongo::NamespaceStringOrUUID const&, mongo::LockMode, mongo::auto_get_collection::OptionsWithSecondaryCollections) ()
h1.6  0x0000aaaaafbf41f8 in mongo::FeatureCompatibilityVersion::findFeatureCompatibilityVersionDocument(mongo::OperationContext*) ()
h1.7  0x0000aaaaafbf68ac in mongo::FeatureCompatibilityVersion::initializeForStartup(mongo::OperationContext*) ()
h1.8  0x0000aaaaae6e5ce8 in mongo::(anonymous namespace)::startupRecovery(mongo::OperationContext_, mongo::StorageEngine_, mongo::StorageEngine::LastShutdownState, mongo::startup_recovery::StartupRecoveryMode, mongo::BSONObjBuilder*) ()
h1.9  0x0000aaaaae6e3f38 in mongo::startup_recovery::repairAndRecoverDatabases(mongo::OperationContext_, mongo::StorageEngine::LastShutdownState, mongo::BSONObjBuilder_) ()
h1.10 0x0000aaaaae69b160 in mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) ()
h1.11 0x0000aaaaae698b9c in mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) ()
h1.12 0x0000aaaaae692e8c in mongo::mongod_main(int, char**) ()
h1.13 0x0000aaaaae692ae0 in main ()

[yota9]

Thanks @peterwaller-arm ! But unforchantly is quite big function and I need to know exact place where the SIGSEGV occured. If you can repeat the process please run it with either set disable-randomization off in GDB or run the app with setarch $(uname -m) -R /path/to/mongod.bolt. Also please give the address of mongo::LockManager::lock in binary just in case e.g. with nm mongod.bolt | c++filt | grep mongo::LockManager::lock cmd. Thank you!

[peterwaller-arm]

unfortunately it's a PIE binary so ASLR is in effect

Correction for clarity: it's not that ASLR is in effect, just that the VMA as reported in the binary is different from where it is loaded at runtime.

To produce the binary I ran llvm-bolt mongod -o mongod.bolt --data perf.boltdata -lite=1 -split-functions on the files attached in the body.

The stack traces of the crash are random and different, here is the first one I hit.

In gdb I did b mongo::transport::SessionWorkflow::Impl::WorkItem::initOperation which reported the address as 0xaaaab2a7af54, and nm reports the address as 0x0807af44 so the offset is 0xaaaaaaa00010.

The crashy code:

   0x0000aaaab2b43b04 <+388>:   adrp    x16, 0xaaaab2acc000 <_ZN5mongo12_GLOBAL__N_110PosixTimerD0Ev+52>
   0x0000aaaab2b43b08 <+392>:   add     x16, x16, #0x978
   0x0000aaaab2b43b0c <+396>:   br      x16
   0x0000aaaab2b43b10 <+400>:   rbit    x17, x16
   0x0000aaaab2b43b14 <+404>:   clz     x17, x17
   0x0000aaaab2b43b18 <+408>:   add     x17, x14, x17, lsr #3
   0x0000aaaab2b43b1c <+412>:   and     x17, x17, x11
   0x0000aaaab2b43b20 <+416>:   ldr     x17, [x12, x17, lsl #3]
   0x0000aaaab2b43b24 <+420>:   ldr     w17, [x17] <----------------------- crash

#0  0x0000aaaab2b43b24 in mongo::ServiceContext::makeOperationContext(mongo::Client*) [clone .cold.0] ()
#1  0x0000aaaab2a7af7c in mongo::transport::SessionWorkflow::Impl::WorkItem::initOperation() ()
#2  0x0000aaaab2a796ac in mongo::transport::SessionWorkflow::Impl::_dispatchWork() ()
#3  0x0000aaaab2a7a448 in _ZZNO5mongo14future_details10FutureImplISt10unique_ptrINS_9transport15SessionWorkflow4Impl8WorkItemESt14default_deleteIS6_EEE4thenINS_19CleanupFuturePolicyILb0EEEZNS5_15_doOneIterationEvE3$_0Q14isFuturePolicyITL0__EEEDaT_OT0_ENKUlOS9_E_clESJ_ ()
#4  0x0000aaaab2a79b80 in mongo::transport::SessionWorkflow::Impl::_doOneIteration() ()
#5  0x0000aaaab2a7a90c in mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::SessionWorkflow::Impl::_scheduleIteration()::$_0>(mongo::transport::SessionWorkflow::Impl::_scheduleIteration()::$_0&&)::SpecificImpl::call(mongo::Status&&) ()
#6  0x0000aaaab2a7bcbc in void mongo::ClientStrand::run<mongo::transport::SessionWorkflow::Impl::_captureContext(mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}::operator()(mongo::Status)::{lambda()#1}>(mongo::transport::SessionWorkflow::Impl::_captureContext(mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}::operator()(mongo::Status)::{lambda()#1}) ()
#7  0x0000aaaab2a7bbe0 in mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::SessionWorkflow::Impl::_captureContext(mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}>(mongo::transport::SessionWorkflow::Impl::_captureContext(mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}&&)::SpecificImpl::call(mongo::Status&&) ()
#8  0x0000aaaab2ab51f0 in mongo::transport::ServiceExecutorSynchronous::SharedState::WorkerThreadInfo::run() ()
#9  0x0000aaaab18bcf64 in mongo::unique_function<void ()>::makeImpl<mongo::transport::ServiceExecutorSynchronous::SharedState::schedule(mongo::unique_function<void (mongo::Status)>)::$_1>(mongo::transport::ServiceExecutorSynchronous::SharedState::schedule(mongo::unique_function<void (mongo::Status)>)::$_1&&)::SpecificImpl::call() ()
#10 0x0000aaaab2ab541c in mongo::unique_function<void ()>::makeImpl<mongo::transport::launchServiceWorkerThread(mongo::unique_function<void ()>)::$_2>(mongo::transport::launchServiceWorkerThread(mongo::unique_function<void ()>)::$_2&&)::SpecificImpl::call() ()
#11 0x0000aaaab18be69c in ?? ()

nm mongod.bolt | c++filt | grep  mongo::ServiceContext::makeOperationContext
000000000745e39c t mongo::ScopeGuard<mongo::ServiceContext::makeOperationContext(mongo::Client*)::$_1>::~ScopeGuard()
00000000080cc680 T mongo::ServiceContext::makeOperationContext(mongo::Client*)
0000000008143980 t mongo::ServiceContext::makeOperationContext(mongo::Client*) [clone .cold.0]
0000000007907910 d mongo::ServiceContext::makeOperationContext(mongo::Client*)::_failedAssertionSourceLocation
000000000745fbd4 t mongo::ServiceContext::makeOperationContext(mongo::Client*)::$_2::operator()[abi:cxx11]() const
000000000745e308 t mongo::ServiceContext::makeOperationContext(mongo::Client*)::$_4::operator()() const

(gdb) info proc mappings
          Start Addr           End Addr       Size     Offset  Perms  objfile
       0x3bc6609a000      0x3bc660aa000    0x10000        0x0  ---p
       0x3bc660aa000      0x3bc660ba000    0x10000        0x0  r-xp
       0x3bc660ba000      0x3bce5c9a000 0x7fbe0000        0x0  ---p
      0xaaaaaaa00000     0xaaaaae8b2000  0x3eb2000        0x0  r--p   mongod.bolt
      0xaaaaae8c1000     0xaaaab2180000  0x38bf000  0x3eb1000  r-xp   mongod.bolt
      0xaaaab218f000     0xaaaab232c000   0x19d000  0x776f000  r--p   mongod.bolt
      0xaaaab233b000     0xaaaab234c000    0x11000  0x790b000  rw-p   mongod.bolt
      0xaaaab234c000     0xaaaab273f000   0x3f3000        0x0  rw-p
      0xaaaab2800000     0xaaaab3430000   0xc30000  0x7e00000  r-xp   mongod.bolt

[yota9]

Thanks @peterwaller-arm ! I can tell that the part of the code you're referring to in asm is shared between many functions, so unfortunately it seems the problem is not in one of the random functions it fails randomly. I would need to understand what it tries to load and how this data supposed to be passed in normal case to understand where the fail occurred :(

[peterwaller-arm]

Is there anything I can do to help facilitate? @paschalis-mpeis did provide binaries, but if you don't want to use those there are also repro instructions which build everything from source and don't use those binaries.

[paschalis-mpeis]

Thanks @yota9 for looking into this, and many thanks to @peterwaller-arm for following up. Currently I'm away with limited access, but I'll be sharing some of the debugging walkthrough that was performed some time ago.

The following observations were made with a similar setup to the reproducer instructions and the attached binary. It may have had slightly different compilation instructions (with full debug symbols) and a different BOLT profile, but it matches @peterwaller-arm's crash point, which is the important part!

---

Debugger walkthrough:

Sample GDB commands to navigate to the bug:

layout split
b mongo::LockManager::lock
r
c 10

# set a breakpoint at each inlined site (831 in total):
b src/third_party/abseil-cpp/dist/absl/container/internal/raw_hash_set.h:1469
# instead of the above I was breaking at a specific address. probably different than the attached binary.
# b *0xaaaab2a62220 

s
c 3 # the 3rd time that this inlined method runs, it hits the **crash**

Relevant assembly extract:

Please read the inline asm comments. The indirect JUMP-04 seems to have landed at a random instruction in the middle of a function. The landing spot was a few instructions before the observed crash.

; [P01]: Jumping to this function: seems OK.
; FUNCTION:
0000000008062100 <_ZN5mongo11LockManager4lockENS_10ResourceIdEPNS_11LockRequestENS_8LockModeE>:
 ; ... ; (omitted earlier/unrelated asm code)
 8062220: f9400232      ldr     x18, [x17]
 8062224: eb14025f      cmp     x18, x20
 ; [P02]: JUMP-01 (direct jump)
 8062228: 54001521      b.ne    0x80624cc <_ZN5mongo11LockManager4lockENS_10ResourceIdEPNS_11LockRequestENS_8LockModeE+0x3cc>
 ; ...
 ; [PO2]: LANDING-OF: JUMP-01 (landing seems OK)
 80624cc: f00006f0      adrp    x16, 0x8141000 <_ZN5mongo25UniqueOperationIdRegistry11acquireSlotEv.cold.0+0xb8>
 80624d0: 91340210      add     x16, x16, #0xd00 ;
 ; [P03]: JUMP-02 (indirect jump)
 80624d4: d61f0200      br      x16 


; [P04]: LANDING-OF: JUMP-02 (landing seems OK)
; FUNCTION:
0000000008141d00 <_ZN5mongo11LockManager4lockENS_10ResourceIdEPNS_11LockRequestENS_8LockModeE.cold.0>:
 8141d00: d1000611      sub     x17, x16, #0x1
 8141d04: ea100230      ands    x16, x17, x16
 ; [P05]: JUMP-03: (NOT taken; direct, conditional jump)
 8141d08: 54000080      b.eq    0x8141d18 <_ZN5mongo11LockManager4lockENS_10ResourceIdEPNS_11LockRequestENS_8LockModeE.cold.0+0x18>
 ; [P06]: (fallthrough)
 8141d0c: b0fff910      adrp    x16, 0x8062000 <_ZN5mongo10LockerImpl10_lockBeginEPNS_16OperationContextENS_10ResourceIdENS_8LockModeE+0x3c0>
 8141d10: 91083210      add     x16, x16, #0x20c
 ; [P07]: JUMP-04 (indirect jump)
 8141d14: d61f0200      br      x16 


; [P08]: LANDING-OF: JUMP-04 at 0x806220c (seems WRONG):
; Landed deep inside the function listed below:
; - does not look intentional.
; - does not make sense how x10 is used below.
; std::__countr_zero<unsigned long> (__x=187650118394380) at
; /bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bit:191
32147582 
; FUNCTION:
0000000008062100 <_ZN5mongo11LockManager4lockENS_10ResourceIdEPNS_11LockRequestENS_8LockModeE>:
 ; ... ; omitted code before the landing instruction
 806220c: dac00211      rbit    x17, x16 ; <- landing instruction
 8062210: dac01231      clz     x17, x17
 8062214: 8b510dd1      add     x17, x14, x17, lsr #3
 8062218: 8a0a0231      and     x17, x17, x10 ; <- x10 had no prior use
 806221c: f8717991      ldr     x17, [x12, x17, lsl #3]
 8062220: f9400232      ldr     x18, [x17] ; <- SIGSEGV. x17 address seems wrong

Follow-up investigation

After the above, the offending function was added to a skip-funcs file so BOLT would not process it, and the whole process was repeated. Iteratively, IIRC I've added more functions to skip-funcs until the crash was no longer observed. Instead, the database froze, waiting on futexes (fast, user-space mutexes).

This freezing was likely due to random jumps in other functions, such as JUMP-04, leading to the observed deadlock.

[yota9]

there are also repro instructions which build everything from source and don't use those binaries.

I'm trying to reproduce the build steps and it fails with 100500 errors like
incomplete type '_Value_type' (aka 'mongo::Value') used in type trait expression or
no matching function for call to '_Destroy' or
arithmetic on a pointer to an incomplete type 'mongo::Value' or
attempt to use a deleted function ¯_(ツ)_/¯

[peterwaller-arm]

I was able to get a build (using repro bits + patch below), but not trivially.

I tried to follow the repro steps and also had build failures, but different from what you describe, which sounds quite a bit worse. I can only speculate why we're seeing different from what @paschalis-mpeis sees. One thing which comes to mind is if we're matching up our llvm/clang commit. The other is if the patch supplied is accidentally incomplete. Another is if the mongo r7.0.5 branch was changed (but this seems unlikely).

For my reproducer bits, I have:

• Ubuntu 22.04.4 LTS on aarch64.
• mongodb/mongo@882248b + below diff
• llvm @ d782119

I had to apply the following patch to mongodb to get things to build (which is quite perplexing, I don't see how this would ever have worked unless some config flag caused these TUs to not be used. The brokenness remains on the master branch):

diff --git a/src/third_party/abseil-cpp/dist/absl/hash/internal/low_level_hash.cc b/src/third_party/abseil-cpp/dist/absl/hash/internal/low_level_hash.cc
index 6f9cb9c7..d288f3ec 100644
--- a/src/third_party/abseil-cpp/dist/absl/hash/internal/low_level_hash.cc
+++ b/src/third_party/abseil-cpp/dist/absl/hash/internal/low_level_hash.cc
@@ -40,7 +40,7 @@ static uint64_t Mix(uint64_t v0, uint64_t v1) {
 }

 uint64_t LowLevelHash(const void* data, size_t len, uint64_t seed,
-                      const uint64_t salt[]) {
+                      const uint64_t salt[5]) {
   const uint8_t* ptr = static_cast<const uint8_t*>(data);
   uint64_t starting_length = static_cast<uint64_t>(len);
   uint64_t current_state = seed ^ salt[0];
diff --git a/src/third_party/boost/boost/thread/future.hpp b/src/third_party/boost/boost/thread/future.hpp
index bbf0ffc8..989babac 100644
--- a/src/third_party/boost/boost/thread/future.hpp
+++ b/src/third_party/boost/boost/thread/future.hpp
@@ -4669,7 +4669,7 @@ namespace detail
       }
       run_it& operator=(BOOST_THREAD_RV_REF(run_it) x) BOOST_NOEXCEPT {
         if (this != &x) {
-          that_=x.that;
+          that_=x.that_;
           x.that_.reset();
         }
         return *this;
diff --git a/src/third_party/mozjs/extract/js/src/threading/ExclusiveData.h b/src/third_party/mozjs/extract/js/src/threading/ExclusiveData.h
index c43cd179..d55cdf37 100644
--- a/src/third_party/mozjs/extract/js/src/threading/ExclusiveData.h
+++ b/src/third_party/mozjs/extract/js/src/threading/ExclusiveData.h
@@ -110,7 +110,7 @@ class ExclusiveData {
       : lock_(id), value_(std::forward<Args>(args)...) {}

   ExclusiveData(ExclusiveData&& rhs)
-      : lock_(std::move(rhs.lock)), value_(std::move(rhs.value_)) {
+      : lock_(std::move(rhs.lock())), value_(std::move(rhs.value_)) {
     MOZ_ASSERT(&rhs != this, "self-move disallowed!");
   }

[paschalis-mpeis]

Given that the below is now merged, the bug described in this issue is no longer triggered:

• [AArch64][BOLT] Ensure tentative code layout for cold BBs runs. #96609

However, it is not entirely clear whether #96609:

• was an improvement to LongJmp, as it avoids expanding some of the stubs,
• or whether it is actually a fix to this issue as it affects more parts of the pass.

To help decide on the status of this issue, I've done some follow-up investigations comparing those areas with and without #96609 applied. Those will be shared soon.

Updated reproducer:

To help with building stage-0 binary from sources (step 1), I've wrapped the previous instructions in a docker script. I've also updated the instructions that will now use the latest trunk version that does not contain the improvement/fix.

[paschalis-mpeis]

Followup observations on the impact of #96609

Below are additional observations regarding the impact of the now-merged improvement/fix on the rest of the pass:

• [AArch64][BOLT] Ensure tentative code layout for cold BBs runs. #96609
• This will be referred to below as 'including the fix'.

Summary

Without the fix there seems to be a cascading effect on stub expansion: way more stubs are created and 2 more iterations are needed to reach a fixed point. There is some evidence to believe that side-effects from such size expansion do not seem to be the issue: a 'fake relaxation' experiment was conducted where binaries ran well up to the point were JITLink failed at static time (ICE crash).

Also, sorting in stub groups becomes flawed, causing undesirable selections from those maps. What might be even worse is that the range-check performed is wrong as it is also based on wrong DotAddresses. As a result, the pass might think a stub is in range while it's not, or vice versa. This may be having some side-effects on the CFG itself but it's unclear if actual errors are corrected before making it to the binary.

Details

1. Number of stubs added:

With the fix, LongJmp inserts 14x less hot and 20x less cold stubs with --group-stubs enabled (see 'Stub Insertion Stats' table at section 'Nop-Experiment Stats'). This by itself shouldn't be causing correctness issues, given surrounding passes/checks work well. Still, it does flood the binary with stubs.

2. Is code size expansion from superfluous stub creation causing problems?

Trying to answer this, I did the below experiment:
I've modified the code before the fix to do a 'fake relaxation' for the stubs that were wrongly going to be relaxed. Instead of relaxing each stub to the ADRP/ADD/BR sequence, some nops were added instead. I tried with a different number of nops (2, 10, 100, 500, 1k). A JITLink static-time error occurred with 1k nops.
All other nop options lead to a successful execution. That is an indication that code expansion was not to blame. ✅

The version with 2 nops in particular, causes an identical fixed-point algo behavior regarding the stub creation & relaxation compared to the code without the fix. The hot/cold region sizes and the hot/cold/shared stub statistics were identical. That is because the same number of instructions were emitted in the binary.

Expand below for more details. Next section shows more details on sizes and stub statistics.

Example of fake relaxation

Apart from emitting the nops, the surrounding code was also patched slightly, so that the exact same stub creations + relaxations happen as with the code without the fix (at least for nops=2, as with higher numbers there is a cascading effect of stub creation). This essentially gives the same sizes and branch distances, as each ADRP+ADD pair is swapped out for a NOP+NOP pair.

(could share this patch upon request)

Example:
Assuming the original stub is the below and was going to be expanded:

b _target

Instead of being expanded to:

adrp    x16, 0xabc000 <_target+0x10>
add     x16, x16, #0x111
br      x16

It is expanded to:

nop
nop
; .. more nops
b _target

3. Nop-Experiment Stats

The statistics in this section may indicate the additional time, space, and code-size output overheads when the fix is not used. The 'Fake NopN' columns relate to the 'fake relaxation' experiment (see previous section).

Stub Insertion Stats:

BOLT Version	Iterations	Hot Stubs	Cold Stubs	Shared	Comments
With fix	3	690	509	0
Without fix	5	9617	10747	5042	baseline
Fake Nop2	5	9617	10747	5042	identical to baseline
Fake Nop10	5	9617	10750	5055
Fake Nop100	5	9621	10770	5086
Fake Nop500	8	9734	10913	5086
Fake-Nop1k	14	9881	11048	5086	JITLink failed: out of range of Branch26PCRel fixup

Hot/Cold section sizes and execution issues:

Shows the code size improvements for the fix and overheads for the 'Fake Nops' experiment.
Diff columns show the difference from baseline in percentage if <100%, otherwise in multiples.
Execution column shows whether execution was performed successfully without any runtime crashes (✔️).

BOLT Version	Hot Size	Cold Size	Hot Size Diff	Cold Size Diff	Execution	Comments
With fix	628Ki	803Ki	-17%	-13%	✔️
Without fix	735Ki	908Ki	-	-	✔️	baseline
Fake Nop2	735Ki	908Ki	0%	0%	✔️	identical to baseline. Stub's instruction length is same with 'real' relaxation.
Fake Nop10	1.01Mi	1.22Mi	37%	34%	✔️
Fake Nop100	4.31Mi	4.91Mi	5.86x	5.4x	✔️
Fake Nop500	19.2Mi	21.3Mi	26x	23x	✔️
Fake Nop1k	NAN	NAN	NAN	NAN	NAN	JITLink failed: out of range of Branch26PCRel fixup

4. Info from code areas affected by wrong tentative addresses

4.1 replaceTargetWithStub with wrong StubCreationAddress parameter:

Without the fix, replaceTargetWithStub runs 26,651 times.
Out of these, Frontier was not null 10,254 times and a seemingly correct tentative StubCreationAddress was used.
For the remaining 16,397 times, Frontier was null and wrong tentative StubCreationAddresses were used. ❌
Those addresses were used as the AtAddress parameter in the call described in the next section (4.2).

With the fix, replaceTargetWithStub runs 1,216 times (22x less).
Frontier was not null in 703 cases with seemingly correct tentative StubCreationAddress.
For the remaining 513 times, Frontier was null, and all StubCreationAddress were also seemingly correct.

4.2 createNewStub with wrong AtAddress parameter

Without the fix it runs 20,364 times.
The tentative AtAddress seemed correct for 9,617 cases while for 10,747 it was wrong. ❌

This means that around half of the stubs were placed in a wrong position inside the stub maps (used for grouping). This makes sorting quite flawed, causing lookups through lookupStubFromGroup to pick a 'sub-optimal' stub (or I could say random stub due to the scale of this issue?).

With the fix it run 1,199 times, all with seemingly correct tentative AtAddress.

4.3 lookupStubFromGroup with wrong DotAddress parameter

Without the fix it runs 29,319 times. The DotAddress appears to be correct 8,552 times, while the rest 20,767 it is wrong. So 2/3 times it is wrong and as explained in the previous case, a suboptimal candidate is picked. ❌
What might be a bigger problem is that the range checks might also be wrong as PCOffset depends on the DotAddress. ❌

I have not established whether such wrong decisions are fixed later on (no ICE crashes occur so they are not 'forcibly caught') or whether this inconsistency makes it with some form to the binary. Either way, at least some CFG changes happen later on as the pass tries to stay 'in sync' with the IR, using fixBranches.

With the fix it runs 600 times with all tentative DotAddresses looking good.

4.4. Other areas that looked OK ✅

I haven't noticed anything suspicious that relates to:

• non-simple functions
• frontiers
• constant islands (none was reported by bolt with the input binary)
• more code areas were tested but are not reported here for conciseness

5. Disabling --group-sharing

Does not fix the original issue.

6. Applying #114015: special case where the closest stub is not used.

Does not fix the original issue.