Trunk 6.1 Macro builds appear to incorrectly compile all package dependencies also for the host when cross-compiling?

[finagolfin]

Is it reproducible with SwiftPM command-line tools: swift build, swift test, swift package etc?

• Confirmed reproduction steps with SwiftPM CLI. The description text must include reproduction steps with either of command-line SwiftPM commands, swift build, swift test, swift package etc.

Description

I just tried cross-compiling swift-foundation for Android AArch64 using a recent trunk 6.1 Oct. 8 snapshot bundle generated by my daily Android CI. While the command below built fine, I noticed it was building a lot of files, and was surprised to find it was building FoundationICU twice, once for the linux x86_64 host and then again for the Android AArch64 target! This appears to be a SwiftPM bug, as I see no reason to build these swift-foundation dependencies like swift-collections and swift-foundation-icu for the host too:

> ls .build/x86_64-unknown-linux-gnu/debug/
_FoundationCShims-tool.build                    ModuleCache                                        SwiftParserDiagnostics-tool.build
FoundationEssentialsTests-tool.build            Modules-tool                                       SwiftParser-tool.build
FoundationEssentials-tool.build                 OrderedCollections-tool.build                      SwiftSyntax509-tool.build
_FoundationICU-tool.build                       plugin-tools-description.json                      SwiftSyntax510-tool.build
FoundationInternationalizationTests-tool.build  _RopeModule-tool.build                             SwiftSyntax600-tool.build                                       FoundationInternationalization-tool.build       SwiftBasicFormat-tool.build                        SwiftSyntaxBuilder-tool.build                                   FoundationMacrosTests-tool.build                SwiftCompilerPluginMessageHandling-tool.build      _SwiftSyntaxCShims-tool.build                                   FoundationMacros-tool                           SwiftCompilerPlugin-tool.build                     SwiftSyntaxMacroExpansion-tool.build
FoundationMacros-tool.build                     SwiftDiagnostics-tool.build                        SwiftSyntaxMacros-tool.build
FoundationMacros-tool.product                   swift_foundationPackageDiscoveredTests-tool.build  SwiftSyntax-tool.build
index                                           swift_foundationPackageTests-tool.build            TestSupport-tool.build                                          InternalCollectionsUtilities-tool.build         SwiftOperators-tool.build

I saw this for the first time months ago, so it should be reproducible with Swift 6.0.1 also.

Expected behavior

Only build the Foundation macros for the linux x86_64 host.

Actual behavior

All package dependencies appear to be compiled twice, whether they're used by the macros or not.

Steps to reproduce

1. Download my Android SDK bundle from the link above.
2. ~/swift-DEVELOPMENT-SNAPSHOT-2024-10-08-a-ubi9/usr/bin/swift sdk install swift-DEVELOPMENT-SNAPSHOT-2024-10-08-a-android-24-0.1.artifactbundle.tar.gz
3. cd swift-foundation
4. ~/swift-DEVELOPMENT-SNAPSHOT-2024-10-08-a-ubi9/usr/bin/swift build --swift-sdk aarch64-unknown-linux-android24

Swift Package Manager version/commit hash

Definitely the Oct. 8 6.1 snapshot, probably the 6.0 release also

Swift & OS version (output of swift --version ; uname -a)

Swift 6.1 cross-compiling from Fedora 40 x86_64 to Android AArch64

[finagolfin]

I just tried to reproduce using the official 6.0.1 toolchain for linux with the 6.0.1 Musl SDK bundle and my 6.0.1 Android SDK bundle, but could not. It appears this is a regression in trunk 6.1.

Pinging @MaxDesiatov and @xedin, who worked on getting macros working when cross-compiling during the summer.

[finagolfin]

I'm able to reproduce this bug with the latest Sep. 25 trunk snapshot of the official Musl SDK bundle too:

> cd swift-foundation
> git checkout swift-DEVELOPMENT-SNAPSHOT-2024-09-25-a
> ~/swift-DEVELOPMENT-SNAPSHOT-2024-09-25-a-ubi9/usr/bin/swift build --swift-sdk aarch64-swift-linux-musl
> du -sk .build/x86_64-unknown-linux-gnu/debug/*
44      .build/x86_64-unknown-linux-gnu/debug/_FoundationCShims-tool.build
40      .build/x86_64-unknown-linux-gnu/debug/FoundationEssentialsTests-tool.build
40092   .build/x86_64-unknown-linux-gnu/debug/FoundationEssentials-tool.build
86816   .build/x86_64-unknown-linux-gnu/debug/_FoundationICU-tool.build
32      .build/x86_64-unknown-linux-gnu/debug/FoundationInternationalizationTests-tool.build
21088   .build/x86_64-unknown-linux-gnu/debug/FoundationInternationalization-tool.build
12      .build/x86_64-unknown-linux-gnu/debug/FoundationMacrosTests-tool.build
56192   .build/x86_64-unknown-linux-gnu/debug/FoundationMacros-tool
1704    .build/x86_64-unknown-linux-gnu/debug/FoundationMacros-tool.build
36      .build/x86_64-unknown-linux-gnu/debug/FoundationMacros-tool.product
62556   .build/x86_64-unknown-linux-gnu/debug/index
784     .build/x86_64-unknown-linux-gnu/debug/InternalCollectionsUtilities-tool.build
40272   .build/x86_64-unknown-linux-gnu/debug/ModuleCache
55048   .build/x86_64-unknown-linux-gnu/debug/Modules-tool
4172    .build/x86_64-unknown-linux-gnu/debug/OrderedCollections-tool.build
2164    .build/x86_64-unknown-linux-gnu/debug/plugin-tools-description.json
7580    .build/x86_64-unknown-linux-gnu/debug/_RopeModule-tool.build
1124    .build/x86_64-unknown-linux-gnu/debug/SwiftBasicFormat-tool.build
4136    .build/x86_64-unknown-linux-gnu/debug/SwiftCompilerPluginMessageHandling-tool.build
552     .build/x86_64-unknown-linux-gnu/debug/SwiftCompilerPlugin-tool.build
1120    .build/x86_64-unknown-linux-gnu/debug/SwiftDiagnostics-tool.build
8       .build/x86_64-unknown-linux-gnu/debug/swift_foundationPackageDiscoveredTests-tool.build
8       .build/x86_64-unknown-linux-gnu/debug/swift_foundationPackageTests-tool.build
1540    .build/x86_64-unknown-linux-gnu/debug/SwiftOperators-tool.build
5044    .build/x86_64-unknown-linux-gnu/debug/SwiftParserDiagnostics-tool.build
11608   .build/x86_64-unknown-linux-gnu/debug/SwiftParser-tool.build
52      .build/x86_64-unknown-linux-gnu/debug/SwiftSyntax509-tool.build
52      .build/x86_64-unknown-linux-gnu/debug/SwiftSyntax510-tool.build
52      .build/x86_64-unknown-linux-gnu/debug/SwiftSyntax600-tool.build
4844    .build/x86_64-unknown-linux-gnu/debug/SwiftSyntaxBuilder-tool.build
12      .build/x86_64-unknown-linux-gnu/debug/_SwiftSyntaxCShims-tool.build
2772    .build/x86_64-unknown-linux-gnu/debug/SwiftSyntaxMacroExpansion-tool.build
1180    .build/x86_64-unknown-linux-gnu/debug/SwiftSyntaxMacros-tool.build
71544   .build/x86_64-unknown-linux-gnu/debug/SwiftSyntax-tool.build
664     .build/x86_64-unknown-linux-gnu/debug/TestSupport-tool.build

[dschaefer2]

I just ran into this in another context. SwiftPM elevates all executables to be products, even those used by build plugins. I'll need to double check if this applies to macros (which are actually executables in the end).

I guess I get why this was done since people wanted to make sure they can run executables even if they aren't declared products? But it definitely causes build pain for executables you don't want to run.

[finagolfin]

That sounds like it has always been the case? Whereas this is a recent regression, as Swift 6.0 fixed building macros when cross-compiling a package, #6950, and it doesn't have this excessive compilation bug, only trunk 6.1 does.

[dschaefer2]

A lot of that code has been "adjusted" recently. At any rate, again, not sure about macros, but I'm definitely seeing it on main with plugin tools.