Passing the `-sdk` flag to the trunk 6.1 compiler on linux has the driver and frontend looking in different directories for the same linux SDK overlay

[finagolfin]

Description

I just tried the following command on linux with the latest Sep. 6 trunk 6.1 snapshot toolchain and it failed for the first time:

> ./swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/bin/swiftc swift/test/Interpreter/hello_toplevel.swift -sdk / -v
Swift version 6.1-dev (LLVM 6f0a61d8ba20c6a, Swift cf2af6809ecd52c)
Target: x86_64-unknown-linux-gnu
/home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/bin/swift-frontend -frontend -c -primary-file /home/finagolfin/swift/test/Interpreter/hello_toplevel.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -sdk / -empty-abi-descriptor -Xcc -working-directory -Xcc /home/finagolfin -resource-dir /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift -module-name hello_toplevel -in-process-plugin-server-path /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift/host/libSwiftInProcPluginServer.so -plugin-path /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift/host/plugins -plugin-path /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/local/lib/swift/host/plugins -o /tmp/TemporaryDirectory.6PBlUB/hello_toplevel-1.o
/home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/bin/swift-autolink-extract /tmp/TemporaryDirectory.6PBlUB/hello_toplevel-1.o -o /tmp/TemporaryDirectory.6PBlUB/hello_toplevel-2.autolink
/home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/bin/clang -pie -Xlinker --build-id -Xlinker -rpath -Xlinker /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift/linux -L /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift/linux -L /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift/linux/x86_64 -L /usr/lib/swift/linux -L /usr/lib/swift/linux/x86_64 /usr/lib/swift/linux/x86_64/swiftrt.o /tmp/TemporaryDirectory.6PBlUB/hello_toplevel-1.o @/tmp/TemporaryDirectory.6PBlUB/hello_toplevel-2.autolink --sysroot / -L /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift/linux -lswiftCore --target=x86_64-unknown-linux-gnu -v -o /home/finagolfin/hello_toplevel
error: link command failed with exit code 1 (use -v to see invocation)
clang version 17.0.0 (https://github.com/swiftlang/llvm-project.git 6f0a61d8ba20c6a2c779c1c9c9a118b55d93dd87)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/bin
Found candidate GCC installation: /lib/gcc/x86_64-redhat-linux/14
Selected GCC installation: /lib/gcc/x86_64-redhat-linux/14
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
clang: error: no such file or directory: '/usr/lib/swift/linux/x86_64/swiftrt.o'

This is with no system Swift installed in /usr. If a system Swift 5.10 happens to be there, the link happens to pass because it uses the 5.10 /usr/lib/swift/linux/x86_64/swiftrt.o instead, but that is the wrong version and could break any time that object file changes. A local Swift 5.10 and a recent 6.0 snapshot don't have this problem, ie they don't look in /usr for an SDK overlay altogether.

There are two problems with the above commands in Swift 6.1:

1. Linking fails because the Swift compiler tells the linker to look for a swiftrt.o that doesn't exist.
2. The swift-frontend looks for the linux SDK overlay Swift modules in -resource-dir /home/finagolfin/swift-DEVELOPMENT-SNAPSHOT-2024-09-06-a-ubi9/usr/lib/swift/ and finds them, but the swift-driver tells clang to look for swiftrt.o from that same linux SDK overlay in /usr/lib/swift/linux/x86_64/ instead, which doesn't exist. In other words, the Swift compiler is looking in completely different directories for files from the same linux SDK.

This all appears to be a consequence of the recent push to change the meaning of certain core compilation flags, as detailed in #73581. Briefly, the goal outlined there is to change the currently used flags of -resource-dir and -sdk, which specify the paths to a platform SDK overlay and platform C/C++ sysroot respectively, to the flags -sdk and -sysroot instead, ie -sdk now points at the platform SDK overlay and -sysroot at the platform C/C++ sysroot.

I am fine with this transition in principle, but we need to handle the actual implementation more carefully or very basic commands like this are going to break without explanation. This particular change was made in swiftlang/swift-driver#1560, which I argued against strongly at that time.

Reproduction

See command above

Expected behavior

Everything compiles fine or user is told what they did wrong.

Environment

Fedora 40 linux x86_64

Additional information

There are two ways we could handle this planned transition, which so far has only been partially implemented for non-Darwin platforms:

a) Tell everyone to use the new set of flags and have things randomly break like this if they use the old flags.
b) Put in some logic to guess what the user is trying to do, then try to either translate the flags based on context or give them better error messages on what they're doing wrong.

If we go with a) then 1. above may be fine, but 2. still shows a problem in the underlying compiler logic.

Ideally, we would also centralize all this platform-specific -sdk logic in one place in the compiler, rather than have different components of the compiler like swift-frontend or the ClangImporter do things differently, which caused #74696.

@compnerd, what do you say? We need a better plan for this transition.

@al45tair, as chair of the Platform Steering Group, perhaps you'd like to weigh in here.