Skip to content

build: cap per-link LTO parallelism at four partitions (ADR-1172) - #1290

Merged
lusoris merged 1 commit into
masterfrom
build/bound-lto-link-parallelism
Sep 5, 2026
Merged

lusoris merged 1 commit into
masterfrom
build/bound-lto-link-parallelism

Conversation

@lusoris

@lusoris lusoris commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

core/meson.build enables LTO for every build (b_lto=true), and GCC's plain -flto means -flto=auto: one lto1-ltrans job per core per link, with no jobserver under Ninja. ninja -j 8 overlaps several of the project's dozens of link targets, so each link spawned 32 partitions. Measured today while three agent builds linked at once: 1-minute load 190 on the 32-core workstation, nine lto1-ltrans processes above 700 % CPU.

ADR-1172: keep LTO on, add b_lto_threads=4 to default_options → -flto=4 (GCC) / -flto-jobs=4 (Clang). A build is now bounded at 4 × ninja -j. Overridable per build dir (-Db_lto_threads=0 = compiler default). docs/development/build-flags.md claimed b_lto defaults to false; corrected, and the new option is documented with its override.

Verified: meson setup core/build core -Denable_cuda=false -Denable_sycl=false → b_lto=True b_lto_threads=4; ninja -t commands | grep -o '\-flto[^ ]*' | sort | uniq -c → 1278 -flto=4, nothing else.

Type

  • build — Meson default option

Checklist

  • Commits follow Conventional Commits.
  • make format && make lint is green locally (pre-commit on every touched file).
  • Unit tests — n/a: build option only; the option value is asserted from meson introspect in the reproducer.
  • Docs in the same PR: docs/development/build-flags.md.
  • ADR: docs/adr/1172-bound-lto-link-parallelism.md + index row.
  • SIMD/GPU, twins, new C sources, breaking change, ffmpeg-patches — n/a (libvmaf build flag only).

Bug-status hygiene (ADR-0165)

  • docs/state.md — no row: build-experience change, no functional defect.

Netflix golden-data gate (ADR-0024)

  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.

Deep-dive deliverables (ADR-0108)

  • Research digest — no digest needed: trivial. The mechanism is GCC's documented -flto=auto default and Meson's b_lto_threads mapping, both quoted in the ADR.
  • Decision matrix — ADR-1172 ## Alternatives considered (LTO off / tell every agent / Ninja jobserver / chosen).
  • AGENTS.md invariant note — no rebase-sensitive invariants: the rebase note below carries the one coupling (b_lto_threads exists because b_lto is on).
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/changed/bound-lto-link-parallelism.md.
  • Rebase note — docs/rebase-notes.md entry (fork-edited hunk of the upstream meson.build).

Reproducer

meson setup core/build core -Denable_cuda=false -Denable_sycl=false
meson introspect core/build --buildoptions | jq -r '.[] | select(.name=="b_lto" or .name=="b_lto_threads") | "\(.name)=\(.value)"'
ninja -C core/build -t commands | grep -oE -- '-flto[=a-z0-9-]*' | sort | uniq -c     # only -flto=4
# before this PR, during `ninja -C core/build -j 8`: ps -eo pcpu,comm --sort=-pcpu | head  → lto1-ltrans ×(cores) per link

🤖 Generated with Claude Code

@lusoris lusoris added this to the 1.0.0 — First release milestone Sep 4, 2026
core/meson.build turns LTO on for every build, and GCC's plain -flto means
-flto=auto: one ltrans job per core per link, with no jobserver under ninja.
Three overlapping agent builds pushed the 32-core workstation to load 190.
b_lto_threads=4 joins default_options (-flto=4 / -flto-jobs=4), bounding a
build at 4 x ninja -j. docs/development/build-flags.md wrongly said b_lto
defaults to false; corrected, and the new option documented with its override.
docs/state.md: no bug row (build-experience change, no functional defect).
@lusoris
lusoris marked this pull request as ready for review September 5, 2026 14:23
@lusoris
lusoris force-pushed the build/bound-lto-link-parallelism branch from 2871c3a to 69a9ff1 Compare September 5, 2026 14:23
@lusoris
lusoris merged commit 89c0954 into master Sep 5, 2026
73 checks passed
@lusoris
lusoris deleted the build/bound-lto-link-parallelism branch September 5, 2026 14:50
lusoris added a commit that referenced this pull request Sep 5, 2026
…lint debt

`core/meson.build` sets `b_lto=true` + `b_lto_threads=4` (ADR-1172), so the
lint job's gcc-14 `meson setup` wrote `-flto=4` into `compile_commands.json`.
clang rejects the numeric form of the flag and reports it as a
`clang-diagnostic-error`, which makes `clang-tidy` exit non-zero on every
file — warning-free or not. `Tidy Changed` has therefore failed every C/C++
pull request since #1290 landed; this PR was the first one to hit it.

Both clang-tidy build steps now configure with `-Db_lto=false`, matching what
the SYCL lint lane already does. The diagnostic set is unchanged: replaying
the same TUs against an `-flto=4`-injected copy of the same
`compile_commands.json` gives identical counts (`core/src/output.cpp` 29 both
ways; `core/src/predict.c` and `core/src/feature/feature_extractor.cpp` 0 both
ways). The same flag also closes a blind spot in `Tidy Ratchet`, whose
diagnostic parser only recognises `path:line:col:` entries and so never saw
the location-less driver error its fail-closed check exists to catch.

`core/src/feature/feature_extractor.h` was the last touched file on this
branch still carrying clang-tidy debt (8 findings). clang-tidy has no compile
command for a header and falls back to `feature_extractor.cpp`'s, so it parses
this C header as C++ and proposes rewrites that would break the ~100 C
translation units which include it. Discharged with one cited, file-scoped
`NOLINTBEGIN` / `NOLINTEND` bracket — the shape ADR-1138 prescribes — covering
`modernize-deprecated-headers`, `modernize-use-using`, `performance-enum-size`
and the include-guard `bugprone-reserved-identifier` family. No declaration,
type or guard was changed, so no includer is affected.

Verification: all four CI-gated files (`feature_extractor.cpp`,
`feature_extractor.h`, `libvmaf.c`, `test_feature_extractor.c`) report 0
warnings, 0 errors and exit 0 under clang-tidy 22.1.8 against a CPU-only
build dir configured like the CI lane. `meson test --suite fast --no-suite gpu`
115/115, `test_feature_extractor` 18/18, Netflix golden gate 271 passed /
12 skipped / 0 failed, and the CUDA reproducer this PR exists for still
scores 82.814059 with `--backend cuda --model version=vmaf_v1.0.16_3d0h`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 5, 2026
Two `Rules` gates caught the previous commit:

- `docs/state.md Gate` (ADR-0334) rejects a placeholder PR reference in an
  inserted row. The `T-TIDY-CHANGED-LTO-FLAG-2026-09-05` row said "This PR
  was the first C/C++ PR after #1290"; it now names PR #1312.
- `Release Script Contract` requires `CHANGELOG.md`'s Unreleased block to
  match `changelog.d/`. Re-rendered with
  `scripts/release/concat-changelog-fragments.sh --write`, which picks up
  `changelog.d/fixed/tidy-changed-lto-flag.md`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 5, 2026
…at all (#1326)

Since PR #1290 merged, the required Tidy Changed and Tidy Ratchet gates failed on
every C/C++ pull request regardless of its diff. core/meson.build's
default_options gained b_lto_threads=4 (ADR-1172) to cap link-time parallelism;
meson renders that as GCC's -flto=4, the tidy lane's gcc-14 build writes it into
compile_commands.json, and clang-tidy parses those entries with clang, which
rejects the argument outright:

    clang: error: unsupported argument '4' to option '-flto='

so every translation unit ended in "1 error generated". Reproduced against a
stock master build: 1257 -flto=4 entries, and clang-tidy on core/src/picture.c
reports "Error while processing".

Both tidy builds now configure with -Db_lto=false. They exist only to emit
compile_commands.json and the generated headers, so LTO buys nothing there, and
the SYCL tidy lane already configured that way for its own toolchain reason.
After the change compile_commands.json holds zero -flto entries and the same
file processes cleanly (459 suppressed, no error).

docs/state.md: T-TIDY-CHANGED-LTO-FLAG-2026-09-05 (Recently closed).

Co-authored-by: Lusoris <lusoris@pm.me>
@lusoris lusoris added the type:build Build system / packaging label Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:build Build system / packaging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant