Parallel frontend (-Z threads=8) results in a different output than single-threaded

[vincentdephily]

I wanted to check if using the threaded compiler produced a different binary:

gix clone https://github.com/BurntSushi/ripgrep/ && cd ripgrep
unset RUSTC_WRAPPER
unset RUSTFLAGS
rm -r target/;time cargo +nightly build -r;cp target/release/rg ./rg.single
export RUSTFLAGS="-Zthreads=1"
rm -r target/;time cargo +nightly build -r;cp target/release/rg ./rg.multi1
export RUSTFLAGS="-Zthreads=8"
rm -r target/;time cargo +nightly build -r;cp target/release/rg ./rg.multi8
sha256sum rg.*

I expected to see this happen:

All binaries should be the same.

Note that when not changing settings, multiple rebuilds yield a binary with the same checksum.

Instead, this happened:

Different binaries, even different sizes. I tried with ripgrep and with my own emlop.

For emlop, I also ran its benchmark (./benches/exec_compare.rs -p e:./emlop.multi,e:./emlop.single -n -r 50) and found a 1% perf regression (but this is a noisy benchmark, I'm sure the difference could have gone either way).

Meta

rustc 1.75.0-nightly (0f44eb32f 2023-11-09)
binary: rustc
commit-hash: 0f44eb32f1123ac93ab404d74c295263ce468343
commit-date: 2023-11-09
host: x86_64-unknown-linux-gnu
release: 1.75.0-nightly
LLVM version: 17.0.4

[bjorn3]

Do you have a global -Ctarget-cpu=native in .cargo/config.toml by any chance? Setting the RUSTFLAGS env var will override this. Do I understand it correctly that you observe a difference between rg.multi1 and rg.multi8 too? That isn't explainable by this.

[lqd]

FWIW I'm seeing the same sizes locally (not the same hashes though).

[vincentdephily]

Good catch, I did have target-cpu=native in my ~/.cargo/config. It could easily explain the perf difference in emlop. After commenting all my local config, I get these hashes:

bcb09910f53f5574f2854f6e22cc3438016130804a2592d4b6987f05fd3519f3  rg.multi1
eaa65abebfb6c483535c69f4cbc229122fe44039ff173882bd00821dba679e2a  rg.multi4
eaa65abebfb6c483535c69f4cbc229122fe44039ff173882bd00821dba679e2a  rg.multi8
bcb09910f53f5574f2854f6e22cc3438016130804a2592d4b6987f05fd3519f3  rg.single

Something is still not right, but it's an interesting pattern.

[bjorn3]

With 2 threads does it match 1 thread or 4 and 8 threads?

[vincentdephily]

I reran all the compiles with various values:

bcb09910f53f5574f2854f6e22cc3438016130804a2592d4b6987f05fd3519f3  rg.single
bcb09910f53f5574f2854f6e22cc3438016130804a2592d4b6987f05fd3519f3  rg.multi1
25ded2ccd720693134a04aacc027c4341ac3bac80cec796fd67efcd945fd1af0  rg.multi2
eaa65abebfb6c483535c69f4cbc229122fe44039ff173882bd00821dba679e2a  rg.multi4
1736d1ebeeedd9fc8750f189b7fc7702452f0daacc04aa62077c61e9c892644b  rg.multi8
1736d1ebeeedd9fc8750f189b7fc7702452f0daacc04aa62077c61e9c892644b  rg.multi10
8d6170a1118a247096a51af01563f714238b1586c6d5dfebe2e2b0dd5add90bd  rg.multi12

-rwxr-xr-x 1 work work 37252584 Nov 10 11:21 rg.single
-rwxr-xr-x 1 work work 37252584 Nov 10 11:22 rg.multi1
-rwxr-xr-x 1 work work 37252584 Nov 10 11:22 rg.multi2
-rwxr-xr-x 1 work work 37252584 Nov 10 11:23 rg.multi4
-rwxr-xr-x 1 work work 37252608 Nov 10 11:23 rg.multi8
-rwxr-xr-x 1 work work 37252608 Nov 10 11:24 rg.multi10
-rwxr-xr-x 1 work work 37252608 Nov 10 11:25 rg.multi12

My CPU has 8 cores, an AMD ryzen 7 4700U.

Note how multi8 got a different hash than multi4 this time around, but still managed to hit the same hash as multi10. Rerunning multi12 a bunch of times, I get a small set of different hashes. So there's a race condition, but a small amount of possible combinations.

[vincentdephily]

Disassembling with objdump -d two files with a different hash, I get a large number of differences in symbol names, but no actual machine code difference:

@@ -205154,29 +205154,29 @@
   133c29:      00 00 00 
   133c2c:      0f 1f 40 00             nopl   0x0(%rax)
 
-0000000000133c30 <_ZN4core3ptr41drop_in_place$LT$rg..subject..Subject$GT$17h57461cc2c7343bc2E.llvm.10780851099308281220>:
+0000000000133c30 <_ZN4core3ptr41drop_in_place$LT$rg..subject..Subject$GT$17h57461cc2c7343bc2E.llvm.8509150623986555565>:
   133c30:      53                      push   %rbx
   133c31:      48 89 fb                mov    %rdi,%rbx
   133c34:      48 83 3f 00             cmpq   $0x0,(%rdi)
-  133c38:      74 18                   je     133c52 <_ZN4core3ptr41drop_in_place$LT$rg..subject..Subject$GT$17h57461cc2c7343bc2E.llvm.10780851099308281220+0x22>
+  133c38:      74 18                   je     133c52 <_ZN4core3ptr41drop_in_place$LT$rg..subject..Subject$GT$17h57461cc2c7343bc2E.llvm.8509150623986555565+0x22>
   133c3a:      48 8b 73 10             mov    0x10(%rbx),%rsi
   133c3e:      48 85 f6                test   %rsi,%rsi
-  133c41:      74 0f                   je     133c52 <_ZN4core3ptr41drop_in_place$LT$rg..subject..Subject$GT$17h57461cc2c7343bc2E.llvm.10780851099308281220+0x22>
+  133c41:      74 0f                   je     133c52 <_ZN4core3ptr41drop_in_place$LT$rg..subject..Subject$GT$17h57461cc2c7343bc2E.llvm.8509150623986555565+0x22>
   133c43:      48 8b 7b 08             mov    0x8(%rbx),%rdi
   133c47:      ba 01 00 00 00          mov    $0x1,%edx
   133c4c:      ff 15 de df 39 00       call   *0x39dfde(%rip)        # 4d1c30 <_GLOBAL_OFFSET_TABLE_+0x920>
   133c52:      83 7b 38 09             cmpl   $0x9,0x38(%rbx)

[lqd]

This is also what I saw, but I had no size differences while you claimed to see some?

[vincentdephily]

Yes I do see some size differences, although nothing as drastic as when target-cpu=native was messing up my analysis.

Just 24 bytes of difference, could it simply be attributed to symbol name length and some padding ?

[lqd]

Yeah maybe; it's hard to tell for sure without being able to reproduce. @bjorn3: in theory the symbol names could even already depend on the number of codegen threads and their schedule, independently of the parallel frontend, or not? (Most/all of the differences in symbol names I'm seeing are in the LLVM hashes for LTO and the likes)

[bjorn3]

AFAIK the codegen unit partitioning ignores -Zthreads entirely.

Most/all of the differences in symbol names I'm seeing are in the LLVM hashes for LTO and the likes

Maybe we forget to sort a list somewhere in the backend causing it to depend on the order in which threads finish?

[lqd]

AFAIK the codegen unit partitioning ignores -Zthreads entirely.

yes this is why I’m saying we may have already been depending on the number of threads/ordering in the backend for some time, even without parallel queries being involved 🤔

[bjorn3]

Without parallel queries we do translation of each codegen unit to llvm ir serially in order and only optimizations in parallel. With parallel queries we do translation in parallel too. Maybe that affects something?