Much higher compile times with `-Z threads=8` than with `-Z threads=1`

[Shnatsel]

When compiling cargo audit from git on commit b6baecc0ea4e2d115e4e10b10c2196b33d42c1da, I'm seeing the project build in 19 seconds on my machine with -Z threads=1 but it takes 25 seconds with -Z threads=8.

I am using a 6-core desktop CPU, so no chance of NUMA issues. I'm also seeing 25s compile times with -Z threads=6, matching the CPU core count.

I've captured Samply profiles but they are hard to make sense of due to the sheer number of threads (4000 for a single thread, 6000 for multiple threads). They are too big for sharing via firefox.dev, so please find them attached:
profile-1-thread.json.gz
profile-8-threads.json.gz

Meta

rustc --version --verbose:

rustc 1.75.0-nightly (fdaaaf9f9 2023-11-08)
binary: rustc
commit-hash: fdaaaf9f923281ab98b865259aa40fbf93d72c7a
commit-date: 2023-11-08
host: x86_64-unknown-linux-gnu
release: 1.75.0-nightly
LLVM version: 17.0.4

[Shnatsel]

Outputs of cargo build --timings:
cargo-timing-1-thread.html.gz cargo-timing-1-thread-no-mold.html.gz
cargo-timing-8-threads.html.gz

[mjguzik]

I confirm the issue, compiling the same thing:

cargo build -r 606.59s user 33.28s system 1596% cpu 40.078 total
RUSTFLAGS="-Z threads=8" cargo build -r 771.61s user 49.88s system 1660% cpu 49.476 total

Test system has 24 cores.

rustc 1.75.0-nightly (0f44eb3 2023-11-09)
binary: rustc
commit-hash: 0f44eb3
commit-date: 2023-11-09
host: x86_64-unknown-linux-gnu
release: 1.75.0-nightly
LLVM version: 17.0.4

[Kobzol]

Do you have the same flags set for both builds? using RUSTFLAGS overrides config.toml, did you perhaps have some options there?

[mjguzik]

In my case this is a fresh clone, 0 local changes.

[Shnatsel]

I did have mold configured as the linker in config.toml. After removing the config the parallel frontend is still slower, although by not quite as much.

Single thread: 23.29s
8 threads: 25.70s

That's still a 10% regression.

[Shnatsel]

I've re-measured cargo build --timings and updated the post above to provide a correct baseline, without mold.

I still see syn v1.0.109 compilation time going from 2.3s to 7.4s, tokio v1.29.1 going up from 2.5s to 5.2s, aho-corasick v1.0.2 from 1.1s to 5.1s, and the compilation time of many other crates also increasing. See the full output of --timings for details.

[mjguzik]

I reran with RUSTFLAGS="-Z threads=1", got:

RUSTFLAGS="-Z threads=1" cargo build -r 591.51s user 30.31s system 1561% cpu 39.825 total

which is about the same as without the -Z flag.