WIP: Add PGO for apple darwin targets

[osiewicz]

Related to #79442

[rust-log-analyzer]

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

info: removing rustup binaries
info: rustup is uninstalled
##[group]Image checksum input
mingw-check-tidy
# We use the ghcr base image because ghcr doesn't have a rate limit
# and the mingw-check-tidy job doesn't cache docker images in CI.
FROM ghcr.io/rust-lang/ubuntu:22.04

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
  g++ \
---

COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py,cpp
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---
#12 4.054   Downloading distlib-0.3.9-py2.py3-none-any.whl (468 kB)
#12 4.061      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 83.3 MB/s eta 0:00:00
#12 4.098 Collecting filelock<4,>=3.12.2
#12 4.101   Downloading filelock-3.18.0-py3-none-any.whl (16 kB)
#12 4.134 Collecting platformdirs<5,>=3.9.1
#12 4.137   Downloading platformdirs-4.3.7-py3-none-any.whl (18 kB)
#12 4.220 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#12 4.412 Successfully installed distlib-0.3.9 filelock-3.18.0 platformdirs-4.3.7 virtualenv-20.31.1
#12 4.412 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#12 DONE 4.5s

#13 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#13 DONE 0.0s
---
DirectMap4k:      114624 kB
DirectMap2M:     5128192 kB
DirectMap1G:    13631488 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py,cpp
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py,cpp
##[group]Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.05s
##[endgroup]
WARN: currently no CI rustc builds have rustc debug assertions enabled. Please either set `rust.debug-assertions` to `false` if you want to use download CI rustc or set `rust.download-rustc` to `false`.
[TIMING] core::build_steps::tool::LibcxxVersionTool { target: x86_64-unknown-linux-gnu } -- 0.218
---
##[endgroup]
[TIMING] core::build_steps::tool::ToolBuild { compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu, tool: "tidy", path: "src/tools/tidy", mode: ToolBootstrap, source_type: InTree, extra_features: [], allow_features: "", cargo_args: [], artifact_kind: Binary } -- 35.638
[TIMING] core::build_steps::tool::Tidy { compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu } -- 0.000
fmt check
Diff in /checkout/src/bootstrap/src/core/build_steps/compile.rs:1395:
                 let clang_rt_dir = get_clang_cl_resource_dir(builder, clang_cl_path);
                 llvm_linker_flags.push_str(&format!("-L{}", clang_rt_dir.display()));
             }
-        } else  {
-
+        } else {
             llvm_linker_flags.push_str("-L/opt/homebrew/Cellar/llvm/20.1.2/lib/clang/20/lib/darwin/lib -lclang_rt.profile_osx");
-
         }
-
     }
 
     // The config can also specify its own llvm linker flags.
Diff in /checkout/src/tools/opt-dist/src/main.rs:112:
     MacCi {
         #[clap(flatten)]
         shared: SharedArgs,
-    }
+    },
 }
 

[lqd]

What improvements are you seeing locally?

Whenever we've tried this, every year or so, using the PGO profiles on osx never worked: almost all functions were reported as not being found in the profile and weren't optimized, so there were no real differences or gains. The same process works on linux and windows, and the functions were definitely in the profile files. It was hard to diagnose and debug in LLVM, and we didn't manage to know why it didn't work at the time.

[osiewicz]

@lqd I am seeing few percents of improvement on Zed and ripgrep; admittedly I did not PGO LLVM itself (this branch is a bit of a mess, hence the WIP). I did not notice any logs for missing data in profiles.
I've tested both ripgrep and Zed with -j1 (to reduce the impact of build graph shape on wall time) - baseline is rustc 1.88.0-nightly (2e6882ac5 2025-05-05).
In case of ripgrep clean build went down from 13.5s to 12.5s. In case of Zed, an incremental build of editor (after touching editor.rs) went down from 24.5s to 23s.

Small aside on incremental scenarios

I didn't look thoroughly into it, but while eyeballing the training process I did not once see any incremental scenario being exercised; looking at the code we do seem to exercise all scenarios, but if that's not the case, that'd explain why the gains for Zed are not that big; our incremental build is heavily bottlenecked by deserialization of incremental build cache, which I imagine would benefit greatly from PGO.

[lqd]

The logs are not shown by default unless one uses something like -Cllvm-args=-pgo-warn-missing-function.

[lqd]

I did not once see any incremental scenario being exercised

rust/src/tools/opt-dist/src/training.rs

Line 112 in d4b119a

init_compiler_benchmarks(env, &["Check", "Debug", "Opt"], &["All"], RUSTC_PGO_CRATES)

should run all scenarios, including incremental ones.

I am seeing few percents of improvement on Zed and ripgrep

It would be good to see the results on the rustc-perf benchmarks (with wall times unfortunately, as we don't have icounts or other hardware counters easily available on the CLI), or https://github.com/Zoxc/rcb

[osiewicz]

Yep, my hunch about incremental scenarios not being ran came from CARGO_INCREMENTAL always being set to 0, but it looks like that's set unconditionally, with an override on the crate being built (which makes sense). source