`rustup`-installed nightly compiler no longer works on MacOS Mojave

Code

I tried this on the latest nightly compiler installed by rustup:

cargo build

I expected to see this happen: My project builds.

Instead, this happened: rustc crashed due to a dynamic library linkage problem.

Version it worked on

It most recently worked on: rustc 1.67.0-nightly (96ddd32c4 2022-11-14).

Version with regression

rustc --version --verbose:

It crashes without giving the version, but so far it affects the rustup toolchains nightly-2022-11-16, nightly-2022-11-17, and nightly (which I assume is the same as nightly-2022-11-17 for now.)

Backtrace

The rustc binary segfaults with the following output:

dyld: Library not loaded: @rpath/librustc_driver-dae4a1cad6347bf8.dylib
  Referenced from: /Users/abuse/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/rustc
  Reason: no suitable image found.  Did find:
	/Users/abuse/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/../lib/librustc_driver-dae4a1cad6347bf8.dylib: cannot load 'librustc_driver-dae4a1cad6347bf8.dylib' (load command 0x80000034 is unknown)
	/Users/abuse/.rustup/toolchains/nightly-x86_64-apple-darwin/bin/../lib/librustc_driver-dae4a1cad6347bf8.dylib: stat() failed with errno=1
	/Users/abuse/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/librustc_driver-dae4a1cad6347bf8.dylib: cannot load 'librustc_driver-dae4a1cad6347bf8.dylib' (load command 0x80000034 is unknown)
	/Users/abuse/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/librustc_driver-dae4a1cad6347bf8.dylib: stat() failed with errno=1
Abort trap: 6

Setting RUST_BACKTRACE=1 does not get further diagnostics, probably because the compiler isn't even being run due to the linkage problem.

Likely cause

I am running MacOS Mojave. Yes, this is quite an old OS release, and no, I'm not intending to upgrade any time soon. It seems likely that whoever is building binaries for the x86_64-apple-darwin target has upgraded their build environment and/or changed some settings, inadvertently causing it to produce binaries which require a newer version of MacOS. (Xcode is unhelpful like that.) I haven't seen any announcement of dropping of support for older MacOS versions.

[thomcc]

We might drop older versions (#104385), but even after that, Mojave should still be supported (Mojave is 10.14, we'd be increasing the requirement to 10.12).

This is likely an unintentional regression from #103929, although I do not know for sure. It could also be related to CI changes, as you mention.

[thomcc]

Apparently we may have bumped the CI runners to macOS 12 around the same time, which is a CI change that is much more likely to be the root cause than that PR.

[BlackHoleFox]

I'll take a look at this one and see if the issue comes from my target cleanup PR. @rustbot claim

[ehuss]

I confirmed with a bisect that it started with #104091 which was the first PR to start using XCode 14.

I'm wondering if it is necessary to start using something like -macos_version_min to the linker?

[BlackHoleFox]

which was the first PR to start using XCode 14.

For my own knowledge going forward, how'd you know that? The GA definition says we're using macos-latest but GitHub's blog says this became macOS 12 back in early October.

I'm wondering if it is necessary to start using something like -macos_version_min to the linker?

Since you figured out what actually introduced it (so not any code changes) I'll look at that instead. man ld isn't helpful telling you what -macos_version_min does that -platform_version wouldn't.

[ehuss]

For my own knowledge going forward, how'd you know that?

I've been tracking the issue in #103044. I updated my system to xcode 14, and that test started failing. I also monitor the GitHub changelog blog which announces these changes.

When GitHub starts changing the default for a -latest runner, it does it over a span of months, migrating individual repos along the way. Our CI (rust-lang-ci/rust) started failing on Monday with that error. We had to expedite the fix to get it working again. There is no advance notice when exactly a particular repo will migrate, it just happens (they do warn you, but it can happen at any time over a span of months).

As for determining which image is being used, you can look at the CI logs. For example, in the build for that PR here: https://github.com/rust-lang-ci/rust/actions/runs/3466400077/jobs/5790206511, at the top of the log is "Set up job" which has an entry "Runner image" which tells you which image is being used (and a link to what software is installed). I used that to verify that our CI had migrated to macos-12 when the builds started failing and someone reported it on Zulip.

[ehuss]

man ld isn't helpful telling you what -macos_version_min does that -platform_version wouldn't.

I think -macos_version_min=10.13 is the same as -platform_version macos 10.13 10.13.

[BlackHoleFox]

I think I found the issue and solution (though the fix is kinda obvious): As of XCode 14, the macOS SDK no longer supports any versions below 10.13 (High Sierra). While I've yet to test this conclusively my thinking is that even though we're telling the linker -platform_version macos 10.7 10.7 its just ignoring us and using the "running on system default", whatever that is.

On a similar note, @thomcc tried running rustc on 10.7 a few weeks ago and ended up getting a linker error as well. This would make sense since even XCode 13 only supports back to 10.9, so rustc hasn't supported back to before 10.9 regardless of what we've claimed for a while now (guess no one uses those for development :) )

So given that we're only going to 10.12 in the ongoing-MCP, we're going to be stuck with XCode 13 for the time being to even support host tools on "older" macOS (deploying back should still work). We'll need to override that in any CI job that produces distributed artifacts. I'll test out the theory tomorrow and probably PR that a rough fix for this repo to "confirm"...