Force enabling SwiftCompilerSources in the build

[eeckstein]

This PR is a follow-up of #73247. Now that SwiftCompilerSources work on all supported platforms, we can make sure the compiler is always built with SwiftCompilerSources.
This means that bootstrapping=off cannot be used anymore.

Also, hosttools is now the default bootstrapping mode on all platforms.
This change also enables SwiftCompilerSources in lldb on Windows (#73322)

Before this PR can be merged, swiftlang/llvm-project#8683 must land in llvm-project.

[eeckstein]

@swift-ci test

[compnerd]

Is this really ready? I thought that we had pieces of the swiftcompilersources disabled on Windows still.

[etcwilde]

We still need to be able to have bootstrapping mode off until we can reliably cross-compile the compiler. Otherwise there is no way to bring up new platforms. (Unless you're okay restricting features to what is available and works with the 5.10 compiler.)

[finagolfin]

Is anybody checking that building with bootstrapping off then using that to build a full Swift compiler still works? For example, the regular bootstrapping modes are no longer checked on CI, and have been broken for months on linux.

[etcwilde]

I have been checking. It takes two build-script invocations to get the complete toolchain built. The first stage 0 compiler doesn't have macros or SwiftCompilerSources and it won't pass the test suite. You use that one as the host-tools compiler to bring up a stage 1 compiler with macros and SwiftCompilerSources support. That one will (or should) pass its tests. Finally, I like to do one more stage 2 build with the stage 1 compiler, which compounds issues in either of the other two. Then I run the tests, validation tests, stress tests, and long tests on that stage 2 compiler.

Unfortunately, Swift 6 doesn't build at the moment since there are some unguarded macros in the stdlib, so you can't get through the stage-0 build, but that should be reversible at the moment. You're right though, we really need some level of testing. Either that a stage-0 bootstrapping=off build works, or that we can build the 5.10 compiler, and then build the latest compiler with the 5.10 compiler or we won't be able to bring up Swift on new platforms (at least until we can cross-compile reliably).

[finagolfin]

I have been cross-compiling the Swift compiler to run natively on Android since Swift 5.2, termux/termux-packages#5134, so works fine for me.

[finagolfin]

Wait, @eeckstein, if Evan is still using the compiler with bootstrapping off, can't we keep that mode around until his cross-compilation changes are in? I don't use it and I'm all for your first change in this patch of changing the default mode to HOSTTOOLS, but why disable being able to turn it off right now?

[eeckstein]

It's still possible to turn if off by not providing a host compiler

[finagolfin]

Oh, I didn't understand your comment below before, you simply turned it into an undocumented internal option.

[eeckstein]

exactly

[eeckstein]

I thought that we had pieces of the swiftcompilersources disabled on Windows still.

Only two small things are disabled: an instruction simplification (#73253) and unit tests (#73252). Both should be fixed soon (cc @ahatanaka, @nate-chandler)

[eeckstein]

We still need to be able to have bootstrapping mode off until we can reliably cross-compile the compiler.

@etcwilde I should have mentioned that this is still possible. There is an escape hatch: bootstrapping mode is turned off if no host compiler is found: https://github.com/apple/swift/blob/c10f5ebdaf268f3b17cb7588d862a0147de7f87d/CMakeLists.txt#L920

[eeckstein]

For bringing swift on a new platform we have two short-term solutions:

1. build the 5.10 compiler on the new platform (as @etcwilde said)
2. don't build with bootstrapping by not providing a host compiler. This is not tested in CI, but as long as we are not breaking this intentionally it should not be hard to get it working - in worst case with a few manual workarounds and fixes.

The long-term solution will be cross compiling. I don't see another option.

[eeckstein]

@swift-ci test

[finagolfin]

The long-term solution will be cross compiling. I don't see another option.

I think it is the solution now, as spelled out on the forum by Doug years ago. Hopefully, Evan's work later this year on retooling the CMake support for cross-compilation will make it easier on those porting to a new platform.

[eeckstein]

@swift-ci test