Decide upon and document policy for what LLVM versions are acceptable for CI LLVM tooling

[Manishearth]

Our c-tiny and js-tiny tests use lld, which requires a pinned rustc (LLVM_COMPATIBLE_NIGHTLY) and LLVM that are compatible with each other.

We have existing documented policy on when MSRV and pinned Rust toolchains can be upgraded. We have no documented policy on what upgrades are allowed for the LLVM version. Updating MSRV can cascade to requiring an updated pinned Rust toolchain and LLVM version, as in #5934.

The rule we have roughly followed in the past is that an updated LLVM must be "widely available". A sufficient condition for that has been if the LLVM is available on LTS APT, GitHub Actions runners, and XCode (using this table ) This is not necessarily a necessary condition, however, and we should have one documented.

XCode does a major update only once a year and their latest version (from September) uses LLVM 17, whereas LLVM 18 has been out and in use since early 2024. Using LLVM 17 locks us to MSRVs from January: the last time I did a Rust upgrade there was a relatively narrow region of Rust versions where the Rust-LLVM coupling worked with external LLVM and we were not needing to use LLVM 18.

Personally, I do not think that the LLVM version used by software that updates once a year and does not necessarily pick up the latest LLVM from 6 months before its latest update is going to always be a workable upper bound for us. It has worked in the past, but such a litmus test can lead to the pinned nightly lagging behind for almost two years¹. I think our MSRV policy is quite well thought out and should not be overridden by the release schedule of software that may end up putting us back on MSRVs that are more than a year and a half old.

(It's worth highlighting, these binaries are used for two FFI tests: test-c-tiny and test-js-tiny)

---

Here's a proposed litmus test for things that are necessary and sufficient for an LLVM version to be considered "widely available". The relevant software binaries we care about are clang, llvm, and lld: all three must be obtainable at the requested version.

• The binaries must be available on APT on the latest Ubuntu LTS
• The binaries must be available on the GitHub actions runners without adding a PPA
• The binaries must be available via XCode or brew.
• For core ICU4X developers on corporate hardware, the binaries must be available in some normal supported way (For Googlers this amounts to "available on gLinux APT", which may lag slightly behind public APT)
• It must be possible to download and install the binaries on Windows (basically always true)

A binary not being available via XCode is dispreferred but not a blocker: if choosing between two LLVM versions that work equally well, pick the one available on latest XCode, however if the newer one enables a desired MSRV (or something else), it is fine for it to not work on XCode.

Footnotes

1. Bear in mind, Rust uses LLVM trunk, not LLVM releases, so an LLVM release's changes will be found in rustc for a fair number of months before the release. ↩

[Manishearth]

Approval requested:

• @Manishearth
• @sffc
• @zbraniecki
• @hsivonen
• @robertbastian

[hsivonen]

LGTM. (Side note: It would be good to get clang and other things that are part of the LLVM distribution to be included as optionally-installable rustup components built from the same LLVM repo revision as the LLVM instance used by rustc.)

[sffc]

For the MSRV policy, we have three tiers: "always OK", "OK with approval", and "always not OK".

If Xcode is shipping a compatible LLVM, then that's the last box for "always OK". However, I would rather not require homebrew if possible. I can totally see an iOS project that wants to use ICU4X and builds with "latest Xcode" as its toolchain, and that's exactly the environment where the LLVM versions need to match up.

Xcode is slow to update, but not that slow. Rust is just far faster.

That said, if we feel that the project has sufficient benefit from updating LLVM to a version that exceeds what is available in Xcode, then that's something the TC could agree to.

[sffc]

For reference, here is the PR where LLVM 19 was added to nightly: rust-lang/rust#127513

It seems they added it basically as soon as they could once the stable version corresponding to the nightly was to be released after the LLVM 19 release. This is extremely early.

As far as Xcode, I agree that it is regrettably a bit behind as we speak. The latest Xcode was released on 11 Dec 2024, but it is using the LLVM released on 28 Nov 2023 (12 month lag). This does appear to be a bit unusual; for example, Xcode 15.0 was released on 18 Sep 2023 and it used LLVM 16.0.0 which was released on 17 Mar 2023 (6 month lag).

Xcode appears to have a major release approximately once per year, but they have minor releases more frequently, and according to the Wikipedia page, they sometimes update LLVM versions in minor releases (e.g. 14.2, last release of 2022, to 14.3, first release of 2023).

[sffc]

One might say, "we can expect most macOS developers to have homebrew installed, so it is OK if ICU4X requires an LLVM available only via homebrew."

Reasons I don't buy this argument:

1. It is very possible to do development without homebrew. On my personal M2 macbook, I installed rustup with the shell script, and I have been able to do everything I want in ICU4X without homebrew.
2. I haven't been able to find definitive data on what proportion of iOS developers or Rust developers have homebrew or macports installed. The best I found is Stack Overflow Survey 2023 which has 33% of developers on macOS and 22% of developers using homebrew. That's a lot, but far from everyone.
3. We benefit from reducing adoption barriers as much as possible, and installing a custom LLVM version, whether via homebrew or otherwise, is a barrier. Developers are busy and if something doesn't work out of the box, and they get distracted by an email or a meeting, they might never come back.
4. As a reference point, the Flutter iOS tutorial makes no reference to homebrew. The only reference to any package manager is CocoaPods which is installable from gem which comes built-in with macOS.
5. Toolchain versions are more disruptive than other things that can be installed from homebrew. You can't simply "brew install clang-18" and be done; you need to change your Xcode build to actually use the new clang version.
6. Mac developers may be less inclined to install third-party software than developers on other platforms.

Okay, this is a bit of a soap box for me. I would rather not get into an extended point-by-point argument.

[Manishearth]

Xcode is slow to update, but not that slow. Rust is just far faster.

It's possible that XCode 16.3 or 16.4 will release with LLVM 18, but it's not guaranteed, and it seems like XCode has in the past been inconsistent about doung two LLVM upgrades per XCode version.

Looking at the dates as far as I can tell we've gotten lucky in the past few years, where we've found versions that work. This matches my experience, almost every time I've done a Rust upgrade for ICU4X I've ended up spending a bunch of time picking a nightly toolchain and realizing that, for the Rust upgrades we've chosen, the range of nightlies that works before we hit LLVM issues is actually quite small! In other words, if we had picked even a slightly newer Rust, we wouldn't have been able to make it work. And I don't think we've been particularly aggressive about updating Rust versions in these scenarios, I tend to be rather conservative when making these PRs.

I've been rather frustrated by this in the past and have even considered proposing we remove the LLVM-linked tests (I think it's just optim5); I'm not fully convinced they provide proportional value.

I would rather not get into an extended point-by-point argument.

FWIW I don't think it's particularly productive to put forth a list of points and then say you don't want to get into an extended point by point argument. But yes, I don't want to either. I'd like to understand priorities here, and not go deep into what the usage of brew is.

---

TLDR: I think my issue is less about brew and more about whether this LLVM version actually meaningfully constrains our clients in the first place

In general, I would care much more about the development environment required for this stuff if it were something we expect most of our users to need. But we don't, not for most users. It's not even something we expect most of our developers to need. This is the development environment for a particularly niche FFI CI tutorial/benchmark, one that I find myself running to need super rarely (and I touch FFI all the time), and I do not think most users will need. In fact I think I've needed to¹ run this job more often when trying to pick versions for a rust upgrade than when doing FFI/size work.

There really are two facets to this and we should be clear about which one we are discussing. Are we looking at these as benchmarks, or tutorials?

As benchmarks, I really do not care if a niche benchmark needs a slightly weirder setup. These benchmarks are nice for understanding the impact of a library change on optimized codesize, but we don't need to do that super often.
Our CI also requires dart, and the recommended way of installing dart on Mac is brew. I'd say test-c-tiny and test-js-tiny are at least as niche as test-dart. It does not seem to be that you feel differently in the context of dependencies of benchmarks/tests, but please let me know if you do.

As tutorials, it gets more interesting. As a tutorial, this is something we are asking some of our users to try. Then the question becomes if that group is niche and if requiring strangeness in the toolchain is a problem.

My overall feeling there is that the folks using the full LLVM-paired optimization will be relatively niche, and, more importantly, they will almost 100% be people who are exerting far more control over their toolchain anyway. This is basically embedded developers, and C toolchains for embedded developers are a huge mess (especially on Mac/XCode, which is basically not commonly used in embedded development anyway). I don't think these developers will be copying what we do in our makefiles, but rather using that as signposts to figure out how to tweak their already tuned toolchain.

When it comes to ICU4X clients who care about size I do not think any of them have expressed interest in tying rustc to LLVM versions. If they were to do so, I suspect they would choose the versions based on what they already use (in the case of monorepo projects, this will almost always be "close to trunk"). Basically all larger Rust-plus-C/C++ embedded projects handle their own toolchains to deal with this; this is a solved problem for them, they do not require our guidance on which Rust/LLVM versions to pick. Pure C/++ embedded projects may not have yet paired toolchains but it's typical for them to have a complex bespoke toolchain setup already, and our guidance may help them figure out what they need to do, but probbaly not be something they would be able to just follow directly².

So, I do not think the LLVM version used in these makefiles is an actual constraint on clients. I could be convinced otherwise, but I think this is my core point of contention: for the users for whom this tutorial is relevant, I do not think the actual version used here constrains them in any meaningful way.

Footnotes

1. Distinguishing "needing to" and "end up running by dint of calling test-c or something" here. I do understand that this CI task test-c-tiny is currently bundled inside test-c so people touching C/JS ffi will typically end up running this task. I'd be in support of splitting them out, I've previously been annoyed by test-c taking a while because it needs to recompile everything under test-c-tiny. ↩

2. We probably should expand the readme's of test-c-tiny and test-js-tiny to talk much more about each step and also what it means to pair LLVM/Rust versions to help these users. ↩

[sffc]

My overall feeling there is that the folks using the full LLVM-paired optimization will be relatively niche, and, more importantly, they will almost 100% be people who are exerting far more control over their toolchain anyway.

Ok, let's drill down on this point.

I don't believe these users are niche or should be considered niche. I want LTO-linked ICU4X to be the standard way app developers include us in their C++/Swift projects.

These can be regular app developers, not just professionals with custom toolchains.

The first value proposition of ICU4X is that we give small binary size. LTO is how we achieve that. Sure, gc_sections and binaryen and other alternatives that don't use LTO give some wins, but LTO gives us more.

Especially since Apple invested a lot in LLVM specifically for use in Xcode, it seems totally reasonable that we consider Xcode to be the standard environment we want to support when it comes to LTO.

[Manishearth]

So, firstly, to make sure we are on the same page: we are not talking about LTO-linked ICU4X. We are talking about cross-language LTO-linked ICU4X, specifically the use of -Clinker-plugin-lto paired with ThinLTO. LTO works just fine on stable Rust paired with any Clang. ThinLTO as well, I believe, though some flags may require nightly. The test also uses a panic-immediate-abort build which is unfortunately nightly only, but doesn't care about clang. -Clinker-plugin-lto is the only part there that cares about clang.

We don't even measure the benefit of -Clinker-plugin-lto in our size benchmarks. If this is to be the recommended way for normal FFI users to use ICU4X, I think we should at least know what the benefits are.

I did a quick benchmark on optim5 with and without linker-plugin-lto on the fixeddecimal test, it's a difference from 33K to 34K. That does not at all line up with "cross-language LTO gives us more". We could remove the cross-language LTO from that test and still get the bulk of the wins of (thin)LTO, panic-immediate-abort, two-phaselinking, gc_sections, and debug symbol stripping, which is what I was referring to in my previous comment about "proposing we remove the LLVM-linked tests". I apologize if that wasn't clear, "LLVM-linked" was specifically referring to the tests which are "linked" to the LLVM version, which is not particularly clear use of terminology in this context 😄 .

"ICU4X has small binary size as a value proposition" is something I quite agree with. I do not think this automatically implies any particular set of flags that we expect users to achieve this with, and if we wish to go down that route I think that needs careful thought and consensus. I also think it needs far more measurements than what we have in tree right now.

I also think that the statement "-Clinker-plugin-lto is the standard way app developers include ICU4X in their C++/Swift projects" is not one that I have considered true for the project so far and would not be in favor of considering true for the project in the future. Not saying that you are proposing that, but in case this discussion goes that way, I'd be against it.

[sffc]

LTO works just fine on stable Rust paired with any Clang. ThinLTO as well, I believe, though some flags may require nightly.

To me "LTO" is synonymous with the Clang procedure that involves the compilation step emitting LLVM bitcode with metadata, and then the linker merging the bitcode from all compilation units.

According to the Clang docs:

In ThinLTO mode, as with regular LTO, clang emits LLVM bitcode after the compile phase. The ThinLTO bitcode is augmented with a compact summary of the module.

gc-sections does something similar but without the LLVM bitcode metadata, so it doesn't do as good of a job. As with anything in this area, mileage will vary, but LTO should "always" be better since it operates with more information.

[Manishearth]

To me "LTO" is synonymous with the Clang procedure that involves the compilation step emitting LLVM bitcode with metadata, and then the linker merging the bitcode from all compilation units.

That's still ambiguous. The linker is invoked in multiple phases in our compilation here, it can be invoked by rustc using LLVM bitcode for Rust, it can be invoked by Clang using LLVM bitcode for Clang, and if you use -Clinker-plugin-lto, the Clang invocation is able to consume Rust's LLVM bitcode as well. It is only this final case where LLVM versions need to be in concordance. Everything else works fine and is a relatively normal set of flags to use.

My statement stands: Regardless of what people use "LTO" to mean, the specific optimization that is causing problems here is -Clinker-plugin-lto. This optimization is a relatively niche one, which does bring benefits, but the benefits it brings are slight compared to just using regular LTO, ideally with two compilation steps and ThinLTO.

(benchmarks from #5948)

# note: LTO builds are bloated unless stripped
# stripped here refers to stripping + gc_sections
# panic-abort here refers to panic-immediate-abort std

-rwxr-x--- 1 manishearth primarygroup   47416 Dec 25 12:03 panic-abort-clang.elf
-rwxr-x--- 1 manishearth primarygroup   47416 Dec 25 12:03 panic-abort-clang-stripped.elf

-rwxr-x--- 1 manishearth primarygroup  133520 Dec 25 12:03 panic-abort-lto-clang-twostep.elf
-rwxr-x--- 1 manishearth primarygroup   34040 Dec 25 12:03 panic-abort-lto-clang-twostep-stripped.elf

-rwxr-x--- 1 manishearth primarygroup   33296 Dec 25 12:09 panic-abort-linker-plugin-lto-clang-twostep.elf
-rwxr-x--- 1 manishearth primarygroup   33296 Dec 25 12:08 panic-abort-linker-plugin-lto-clang-twostep-stripped.elf

gc-sections/stripping + two-step ThinLTO provides almost all the benefits available to LTO. Cross language LTO is providing a 2% improvement here.

If this discussion were about choices made by ICU4X that prevent this 2% codesize win I would definitely consider it worth thinking about. But this discussion is about choices made by ICU4X that make this codesize win a little bit harder to achieve on certain platforms. Other codesize win options are already hard to achieve, build-std requires nightly (many, many clients have a distaste of Rust nightly) and we have experienced build-std breakage in the past. I haven't properly benchmarked it but I suspect the build-std-panic-immediate-abort option gets a roughly similar order of magnitude of codesize win in some cases.

(n.b. It's unclear to me if we should always advocate for ThinLTO: The point of using ThinLTO is link-time performance, not binary size, and my benchmarks here don't measure thin vs regular but instead unconditionally use thin for two-step. We may wish to check that at some point, but that's not the core issue in this discussion so I haven't drilled deeper)