Yams fails to build on 5.6.2 with autolink-extract error

[melizasw]

Describe the bug
When building Swift on Debian testing the Building Swift Driver dependency: Yams step fails with:

error: autolink-extract command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: error opening input file \'lib/libYamsTests.a\' (The file was not recognized as a valid object file)
ninja: build stopped: subcommand failed.

Steps To Reproduce
Steps to reproduce the behavior:

1. Obtain Swift 5.6.2 source code from various and sundry git repos.
2. Run swift/utils/build-script --preset=buildbot_linux,no_test install_destdir=$PWD installable_package=$PWD/swift_5.6.2-1-debian.tar.gz

Expected behavior
I expect Yams to build without error.

Environment (please fill out the following information)
Debian testing (bookworm)

Additional context
I did not have this problem when building 5.5.1. This might be related to #57323, but modifying GenericUnixToolchain+LinkerSupport.swift in the "case .staticLibrary:" section to match 5.5.1 and 5.5.2 did not help.

$ ar t build/buildbot_linux/earlyswiftdriver-linux-x86_64/release/dependencies/yams/lib/libYamsTests.a
ConstructorTests.swift.o
EmitterTests.swift.o
EncoderTests.swift.o
MarkTests.swift.o
NodeTests.swift.o
PerformanceTests.swift.o
RepresenterTests.swift.o
ResolverTests.swift.o
SpecTests.swift.o
StringTests.swift.o
TestHelper.swift.o
TopLevelDecoderTests.swift.o
YamlErrorTests.swift.o
YamsTests.autolink

I have not tried -disallow-use-new-driver yet as I need to figure out where to supply that parameter and what impact it will have on the build.

[melizasw]

I've made a little progress on this. I noticed that building Yams resulted in a call to swiftc (5.5.1 on my system):

Building Swift Driver dependency: Yams
[1/10][ 10%][0.100s] Building C object Sources/CYaml/CMakeFiles/CYaml.dir/src/writer.c.o
[2/10][ 20%][0.121s] Building C object Sources/CYaml/CMakeFiles/CYaml.dir/src/reader.c.o
[3/10][ 30%][0.365s] Building C object Sources/CYaml/CMakeFiles/CYaml.dir/src/parser.c.o
[4/10][ 40%][0.445s] Building C object Sources/CYaml/CMakeFiles/CYaml.dir/src/api.c.o
[5/10][ 50%][1.000s] Building C object Sources/CYaml/CMakeFiles/CYaml.dir/src/emitter.c.o
[6/10][ 60%][1.286s] Building C object Sources/CYaml/CMakeFiles/CYaml.dir/src/scanner.c.o
[7/10][ 70%][1.374s] Linking C static library lib/libCYaml.a
[8/10][ 80%][8.809s] Linking Swift static library lib/libYams.a
[9/10][ 90%][17.220s] Linking Swift static library lib/libYamsTests.a
[10/10][100%][17.462s] Linking Swift executable bin/YamsTestRunner
FAILED: bin/YamsTestRunner Tests/CMakeFiles/YamsTestRunner.dir/LinuxMain.swift.o swift/YamsTestRunner.swiftmodule 
: && /usr/bin/swiftc -output-file-map Tests/CMakeFiles/YamsTestRunner ...

I uninstalled swift with sudo dpkg -r swiftlang and now the build process is completely different as Yams is not built early on and the flow looks a lot more like what I saw with 5.5.1 where Yams is built late in the flow using the freshly compiled swiftc binary. So apparently you can't have Swift installed when building Swift?

[etcwilde]

There was a bug in the driver. It should be fixed in 5.7. I remember the PR a couple years back, but I don't remember which one. @compnerd, @artemcm, do either of you remember which PR it was?

[compnerd]

Maybe, possibly, swiftlang/swift-driver@c1332a6? If so, we can pull it into the 5.6 branch in case we end up doing a 5.6 release.

[etcwilde]

Hmm, it should be swiftlang/swift-driver#773, which looks like it's that commit. Unfortunately, that commit is on the driver release/5.6 branch with a pinch more for LTO.

[finagolfin]

So apparently you can't have Swift installed when building Swift?

Rather, the docs explicitly tell you not to build the early swift driver on linux. That's only because we didn't want to make installing a host Swift toolchain a requirement though, not because of any big bugs.

However, small bugs like this have existed in the past: the problem here is that you are building the early swift driver with the old Swift 5.5.1, when the fix for the static library bug you linked was not added till 5.5.2. So, this could be easily fixed by updating your system Swift to 5.5.3, or if that's not easy, simply get the official download of Swift 5.6.2 for linux, extract it locally, and add it to your path.

[melizasw]

Thanks @buttaface. What threw me off is that the build flow changes based on if swift is installed or not and I wasn't considering that as a variable when troubleshooting. I've been leveraging the work of tachoknight's packaging swift for Fedora and am using the same build process so I knew it must be possible somehow. I was also operating under the poor assumption that using --preset=buildbot_linux,no_test would have resulted in a successful linux build without having to also pass --skip-early-swift-driver or any other tweak.

I am packaging swiftlang for Debian so I can't install a version of swift from elsewhere to help the build. My build dependencies must come from the official Debian repo. Otherwise, that would probably work. Now that I have the build working again my biggest issues are actually getting everything into compliance with Debian policies. 😵‍💫

[finagolfin]

I was also operating under the poor assumption that using --preset=buildbot_linux,no_test would have resulted in a successful linux build without having to also pass --skip-early-swift-driver or any other tweak.

The early swift driver is the one part of the build that acts differently based on auto-detection of a host toolchain, maybe that should be made more explicit, @artemcm?

I am packaging swiftlang for Debian so I can't install a version of swift from elsewhere to help the build. My build dependencies must come from the official Debian repo.

Oh, great, well, just so you know, a prebuilt host toolchain will soon be a requirement for upcoming versions of the Swift compiler, as more of it will be implemented in Swift.

Now that I have the build working again my biggest issues are actually getting everything into compliance with Debian policies.

Open a separate Debian packaging issue here and feel free to ask any questions you have.

[etcwilde]

@melizasw, We have some work in that direction. I'm not sure exactly how far along it is or what state things are in but here's a link to the ubuntu focal DEB package from what I can tell; https://github.com/apple/swift-installer-scripts/tree/main/platforms/Linux/DEB/Ubuntu/focal

[melizasw]

@etcwilde Thanks for that. I can see things in your rules file that I spent hours to figure out on my own and know exactly why you have them there. I am not familiar with the Ubuntu packing process and policies. Do you also use debuild?

In your clean section try using py3clean . to clean up after python3.

I'll have to dig through and see what else I can find. I don't think it worthwhile to add what I am going to the repo as it serves one purpose only; to get Swift into the Debian "main" repo. The only place that makes sense is on the Debian salsa server. But if it might help you then I would be happy to upload what I have.

[melizasw]

I can confirm that this error building Yams goes away when the installed switfc is version 5.6.2.

[ephemer]

@buttaface @compnerd I'm tagging you here because I see you've commented on this thread and it seems related (might be wrong though).

We get a similar error building with CMake using @buttaface's Android SDK builds with the 5.7.0 toolchain: unableToFind(tool: "swift-autolink-extract")

We are working around this by adding a symlink into the toolchain: sudo ln -s swift /Library/Developer/Toolchains/swift-5.7-RELEASE.xctoolchain/usr/bin/swift-autolink-extract, which works, but since swift-autolink-extract is apparently not provided any more it seems like this workaround may not work forever. It also makes setting up the Android toolchain more difficult, because it needs sudo access to modify the pre-installed toolchain.

Do either of you have any ideas why this error would occur? I will try to do a SwiftPM build for Android without the extra symlink and will update this post with the result of whether it also fails. Update: it's not just with CMake, it's also SwiftPM builds:

$ swift package init

$ echo '{
        "version": 1,
        "target": "aarch64-unknown-linux-android24",
        "toolchain-bin-dir": "/Library/Developer/Toolchains/swift-5.7-RELEASE.xctoolchain/usr/bin",
        "sdk": "/usr/local/ndk/25.1.8937393/toolchains/llvm/prebuilt/darwin-x86_64/sysroot",
        "extra-cc-flags": ["-fPIC"],
        "extra-swiftc-flags": [
            "-resource-dir",
            "/path/to/sdk/arm64-v8a/usr/lib/swift",
            "-tools-directory",
            "/usr/local/ndk/25.1.8937393/toolchains/llvm/prebuilt/darwin-x86_64/bin"
        ],
        "extra-cpp-flags": ["-lstdc++"]
}' > target.json

$ /Library/Developer/Toolchains/swift-5.7-RELEASE.xctoolchain/usr/bin/swift build --destination target.json

Building for debugging...
error: unableToFind(tool: "swift-autolink-extract")%

[finagolfin]

Yours is a separate problem, a known issue with using the Mac compiler to cross-compile for ELF platforms, which should be fixed in the next release, #60023.