Always build SwiftDriver and use it as default compiler driver #36377

artemcm · 2021-03-09T22:51:27Z

Always build (and use) the new SwiftDriver as the default compiler driver.
This will make sure that compiler developers are using the new driver when they build the compiler locally and use it.

Adds a new build-script product category: before_build_script_impl for products we wish to build before the impl products.
Adds a new EarlySwiftDriver product to that category, which gets built with the host toolchain.
- Adds an escape hatch: --skip-early-swift-driver
Adjusts the swift CMake configuration with an additional step: swift_create_early_driver_symlinks which (if one was built) creates symlinks in the swift build bin directory to the EarlySwiftDriver swift-driver and swift-help executables.
Adds a new test subset : only_early_swiftdriver, which will get built into a corresponding CMake test target: check-swift-only_early_swiftdriver-* which runs a small subset of driver-related tests against the Early SwiftDriver.
- This subset is run always when the compiler itself is tested (--test is specified)
  - With an escape disable-switch: --skip-test-early-swift-driver
- All tests outside of only_early_swiftdriver are forced to run using the legacy C++ driver (to ensure it gets tested, still).
- NOTE: SwiftDriver product (no 'Early') is still the main product used to build the driver for toolchain installation into and for executing the product's own tests. This change does not affect that.

artemcm · 2021-03-10T20:54:31Z

@swift-ci please test

swift-ci · 2021-03-10T21:02:01Z

Build failed
Swift Test Linux Platform
Git Sha - 5f373f14595a8d5fdbba9df10e37b9168397d794

swift-ci · 2021-03-10T21:31:50Z

Build failed
Swift Test OS X Platform
Git Sha - 5f373f14595a8d5fdbba9df10e37b9168397d794

artemcm · 2021-03-11T17:51:22Z

@nkcsgexi curious what you think of this approach

nkcsgexi · 2021-03-11T20:39:50Z

This is a great idea to enable compiler engineers to always test swift-driver as they do for the C++ driver. Thanks for working on this!

artemcm · 2021-03-12T18:53:50Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test

utils/swift_build_support/swift_build_support/products/swiftdriver.py

artemcm · 2021-03-12T22:27:32Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test Linux platform

artemcm · 2021-03-16T18:18:14Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test

artemcm · 2021-03-16T22:33:28Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test macOS Platform

artemcm · 2021-03-17T17:57:26Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test macOS Platform

artemcm · 2021-03-19T16:18:41Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test macOS Platform

artemcm · 2021-03-19T16:25:01Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test macOS Platform

artemcm · 2021-03-19T17:34:21Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test macOS Platform

artemcm · 2021-03-25T20:16:21Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test macOS Platform

artemcm · 2021-03-29T21:50:25Z

Together with: swiftlang/swift-driver#537
@swift-ci please smoke test macOS Platform

atrick · 2021-04-26T15:33:18Z

I'm not sure if my questions even make sense, so you don't need to repeat them verbatim in a doc. I'm confused about the design of the command line interfaces for our driver-like tools and not sure where to look.

artemcm · 2021-04-26T15:47:38Z

Thanks for adding all the build-script info to the docs! Looks good to me.

I'm mostly curious about how to understand/debug differences between the old/new driver and other driver-like tools.

Will 'swiftc ... -###' always print the same set of commands for the old/new driver? Do they call 'swift-frontend' the same way?

The new swift-driver (Still invoked as swiftc) is meant to b a direct drop-in replacement. As such it has the exact same command-line interface. The expectation is that its behaviour closely matches the legacy driver; however, during, and after the transition to the new driver being the default its behaviour may start to diverge from the legacy driver as par for the course of its evolution and gaining new features, etc. Today, broadly-speaking, swift-frontend invocations generated by the two drivers are expected to be very similar.

What's the difference between invoking 'swiftc' vs. 'swift-driver' at the top level?

The mechanism by which we are deploying the new driver today is as follows:

The user invokes swiftc, which is the same executable as before, which forwards the invocation to the swift-driver executable if one is found alongside it. There are two escape hatches that will make swiftc avoid forwarding to the new driver:
-disallow-use-new-driver command line flag
SWIFT_USE_OLD_DRIVER environment variable
These are meant to allow for a fallback to the legacy driver during the transition.
If the user is to directly invoke the swift-driver executable, the behaviour should be the same as invoking swiftc executable, but without the option for a legacy driver fallback.

What's the difference between invoking 'swift -frontend' vs. 'swift-frontend' (I sure hope its nothing)?

AFAIK, there is no difference in doing this with either driver.

If there's a 'swift -frontend' to invoke 'swift-frontend' why isn't there a 'swift -c' to invoke 'swift-driver'?

swift -frontend bypasses any and all driver logic and invokes the compiler directly. With the new driver, once we go through this transition, swift and swiftc executables themselves will be the new driver executables (symlinks to swift-driver). So it seems that swift -c is a bit redundant. Unless I misunderstood this part?

These don't need to be answered in this PR, but I didn't happen to see an explanation in the docs and these are the important questions to me.

@atrick, thank you for these questions, I do think they highlight some pretty obvious gaps in documentation in this transition which must be documented, either in the docs here, or in the swift-driver repo, or both.

I like @gottesmm 's idea of having an F.A.Q. in the docs here, with the swift-driver's own documentation (in its corresponding repository) providing more in-depth docs.

artemcm · 2021-04-26T16:45:27Z

@swift-ci please test

swift-ci · 2021-04-26T18:10:36Z

Build failed
Swift Test Linux Platform
Git Sha - 750923a082acb0de11822c5f269436acdae3a718

swift-ci · 2021-04-26T18:24:19Z

Build failed
Swift Test OS X Platform
Git Sha - 750923a082acb0de11822c5f269436acdae3a718

…iver. This will make sure that compiler developers are using the new driver when they build the compiler locally and use it. - Adds a new build-script product category: before_build_script_impl for products we wish to build before the impl products. - Adds a new EarlySwiftDriver product to that category, which gets built with the host toolchain. - Adds an escape hatch: --skip-early-swift-driver - Adjusts the swift CMake configuration with an additional step: swift_create_early_driver_symlinks which (if one was built) creates symlinks in the swift build bin directory to the EarlySwiftDriver swift-driver and swift-help executables. - Adds a new test subset : only_early_swiftdriver, which will get built into a corresponding CMake test target: check-swift-only_early_swiftdriver-* which runs a small subset of driver-related tests against the Early SwiftDriver. - This subset is run always when the compiler itself is tested (--test is specified) - With an escape disable-switch: --skip-test-early-swift-driver - All tests outside of only_early_swiftdriver are forced to run using the legacy C++ driver (to ensure it gets tested, still). NOTE: SwiftDriver product (no 'Early') is still the main product used to build the driver for toolchain installation into and for executing the product's own tests. This change does not affect that.

…d procedure.

artemcm · 2021-04-26T19:25:15Z

@swift-ci please test

atrick · 2021-04-26T20:35:31Z

Thanks Artem!

I think you can just cross reference Driver.md from this "debugging" document. Explaining the command line interface probably goes better in the driver docs.

swift -frontend bypasses any and all driver logic and invokes the compiler directly. With the new driver, once we go through this transition, swift and swiftc executables themselves will be the new driver executables (symlinks to swift-driver). So it seems that swift -c is a bit redundant. Unless I misunderstood this part?

swift -c is (I think) redundant with swiftc. My point is that swift -frontend is also (I think) redundant with swift-frontend. Is there a reason for the redundancy in one case and not the other. I don't have an opinion, I'm just trying to understand the CLI.

Also, swift has a different CLI from swiftc. So which one is the driver interface?

artemcm · 2021-04-26T22:04:48Z

swift -c is (I think) redundant with swiftc. My point is that swift -frontend is also (I think) redundant with swift-frontend. Is there a reason for the redundancy in one case and not the other. I don't have an opinion, I'm just trying to understand the CLI.

That's a good point, it is redundant, and is just an artifact of the times before swift-frontend. The only thing keeping it around is that we may still have compiler clients who may be relying on it, e.g. things like SwiftPM may still be generating swift -frontend invocations. We should be able to deprecate it if we make sure the clients know to invoke swift-frontend directly.

Also, swift has a different CLI from swiftc. So which one is the driver interface?

Both are the driver, just two different "modes" of the driver. For swift, the driver's job is simply to create the swift-frontend -interpret job and setup its arguments correctly. For swiftc, he driver's job is to plan the build, as normal. The driver, internally, distinguishes them by knowing whether or not it was invoked as swift or swiftc. (or --driver-mode=[swift|swiftc] if invoking the swift-driver directly).

artemcm · 2021-04-26T22:05:17Z

@swift-ci please test Windows platform

In the original implementation (swiftlang#36377), using `--infer` accidentally disables the `earlyswiftdriver` product (`before_impl_product_classes` set to always empty). This change fixes that and makes sure early SwiftDriver is always built, regardless of whether or not `--infer` is used. This change also ensures that `install_all` setting triggered by `--infer` does not affect products which specify `is_ignore_install_all_product` to return True. This is useful for products which should not be installed into the toolchain (corresponding build products that use the just-built toolchain are the products that get installed, e.g. `swiftdriver` to `earlyswiftdriver`).

artemcm force-pushed the NewDriverDefault branch from 5f373f1 to 4513ec2 Compare March 11, 2021 19:44

artemcm force-pushed the NewDriverDefault branch 2 times, most recently from 7d7f6a0 to f9935f6 Compare March 12, 2021 18:51

artemcm commented Mar 12, 2021

View reviewed changes

utils/swift_build_support/swift_build_support/products/swiftdriver.py Outdated Show resolved Hide resolved

artemcm force-pushed the NewDriverDefault branch from f9935f6 to 6084170 Compare March 16, 2021 18:15

artemcm force-pushed the NewDriverDefault branch 2 times, most recently from a10b822 to 4277506 Compare March 16, 2021 21:51

artemcm force-pushed the NewDriverDefault branch from 4277506 to 68ac381 Compare March 17, 2021 17:57

artemcm force-pushed the NewDriverDefault branch 6 times, most recently from 9beccee to 98ad24f Compare March 29, 2021 21:49

artemcm force-pushed the NewDriverDefault branch from 98ad24f to e94617e Compare March 30, 2021 17:09

artemcm added 2 commits April 26, 2021 12:17

Add documentation about the compiler's standalone driver and its buil…

60a4af3

…d procedure.

artemcm force-pushed the NewDriverDefault branch from 750923a to 60a4af3 Compare April 26, 2021 19:19

gottesmm approved these changes Apr 27, 2021

View reviewed changes

artemcm merged commit 23452d5 into swiftlang:main Apr 27, 2021

This was referenced Apr 28, 2021

Revert "Always build SwiftDriver and use it as default compiler driver" #37114

Merged

Revert "Merge pull request #37114 from apple/revert-36377-NewDriverDefault" #37122

Merged

artemcm deleted the NewDriverDefault branch May 7, 2021 16:42

artemcm mentioned this pull request May 10, 2021

[Build Script] Fix --infer with earlyswiftdriver. #37349

Merged

finagolfin mentioned this pull request Nov 2, 2023

[Build] Enable building the early Swift Driver on linux with the prebuilt host toolchain #69564

Merged

Always build SwiftDriver and use it as default compiler driver #36377

Always build SwiftDriver and use it as default compiler driver #36377

Uh oh!

Conversation

artemcm commented Mar 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

artemcm commented Mar 10, 2021

Uh oh!

swift-ci commented Mar 10, 2021

Uh oh!

swift-ci commented Mar 10, 2021

Uh oh!

artemcm commented Mar 11, 2021

Uh oh!

nkcsgexi commented Mar 11, 2021

Uh oh!

artemcm commented Mar 12, 2021

Uh oh!

Uh oh!

artemcm commented Mar 12, 2021

Uh oh!

artemcm commented Mar 16, 2021

Uh oh!

artemcm commented Mar 16, 2021

Uh oh!

artemcm commented Mar 17, 2021

Uh oh!

artemcm commented Mar 19, 2021

Uh oh!

artemcm commented Mar 19, 2021

Uh oh!

artemcm commented Mar 19, 2021

Uh oh!

artemcm commented Mar 25, 2021

Uh oh!

artemcm commented Mar 29, 2021

Uh oh!

atrick commented Apr 26, 2021

Uh oh!

artemcm commented Apr 26, 2021

Uh oh!

artemcm commented Apr 26, 2021

Uh oh!

swift-ci commented Apr 26, 2021

Uh oh!

swift-ci commented Apr 26, 2021

Uh oh!

artemcm commented Apr 26, 2021

Uh oh!

atrick commented Apr 26, 2021

Uh oh!

artemcm commented Apr 26, 2021

Uh oh!

artemcm commented Apr 26, 2021

Uh oh!

Uh oh!

artemcm commented Mar 9, 2021 •

edited

Loading