[Build Script Helper] On macOS make install action use CMake and install libSwiftDriver #246

artemcm · 2020-09-11T00:25:18Z

Previously only the x86 swift-driver and swift-help executables were copied over to the specified toolchain's bin directory.
With this change, libSwiftDriver shared library is also packaged and installed into the toolchain, as well as libSwiftOptions.
To do so, we build the driver and all of its dependencies using CMake for arm64 and x86_64 and create universal binaries, as documented here.

Because the driver's dependencies need to come along for the ride, the installed artifacts are as follows, within the specified toolchain:

Executables:

bin/swift-driver
bin/swift-help

Binary Modules and Shared Libraries

lib/swift/macosx/libTSCBasic.dylib
lib/swift/macosx/libTSCLibc.dylib
lib/swift/macosx/libTSCUtility.dylib
lib/swift/macosx/libSwiftDriver.dylib
lib/swift/macosx/libSwiftOptions.dylib

lib/swift/macosx/TSCBasic.swiftmodule
lib/swift/macosx/TSCLibc.swiftmodule
lib/swift/macosx/TSCUtility.swiftmodule
lib/swift/macosx/SwiftDriver.swiftmodule
lib/swift/macosx/SwiftOptions.swiftmodule

C Module .modulemap and header files

include/swift/TSCclibc/*

Misc

The Yams dependency is implementationOnly and is built as a static-library.
ArgumentParser.swiftmodule and libArgumentParser.dylib must also be installed into the toolchain, but the code currently omits them, to avoid adding them to the toolchain until they are actually needed.

Resolves rdar://69708243

artemcm · 2020-09-15T19:35:58Z

Two blockers for this, currently:

Building swift-driver with library-evolution enabled fails because the compiler prints unqualified types for type extensions to the interface file and we are extending TSCBasic.Process, which is ambiguous with Foundation.Process. Brent has a fix PR to make the compiler print qualified extension type names here: Qualify the names of extended types in module interfaces swift#32780, but just throwing this fix in I am seeing other failures. More investigation needed.
Without library-evolution, the installation works for me locally, but does not work in Swift CI (e.g. here: [Build Script] Pass down a CMake path to swift-driver's build script swift#33918) because when building LLBuild, my local build figures out to use Xcode SDK's version of libSQLite, whereas in CI, the build uses the one in /usr/local/lib/, which only has an x86_64 slice so the arm64 build fails.

nkcsgexi · 2020-09-15T21:21:16Z

@artemcm Just discussed @DougGregor about this, we could first disable -enable-library-evolution for now and use the binary module format for submission. The build system can link against libSwiftDriver in the default toolchain so we could start experimenting with the explicit module feature. Multiplexing libSwiftDriver can be a concern we tackle some point later.

artemcm · 2020-09-15T21:23:23Z

@artemcm Just discussed @DougGregor about this, we could first disable -enable-library-evolution for now and use the binary module format for submission. The build system can link against libSwiftDriver in the default toolchain so we could start experimenting with the explicit module feature. Multiplexing libSwiftDriver can be a concern we tackle some point later.

Sounds good! I'll push through with resolving the remaining issues without using -enable-library-evolution.

nkcsgexi · 2020-09-15T21:24:54Z

@artemcm Just discussed @DougGregor about this, we could first disable -enable-library-evolution for now and use the binary module format for submission. The build system can link against libSwiftDriver in the default toolchain so we could start experimenting with the explicit module feature. Multiplexing libSwiftDriver can be a concern we tackle some point later.

Sounds good! I'll push through with resolving the remaining issues without using -enable-library-evolution.

Thank you!

artemcm · 2020-09-17T00:08:37Z

This is currently functional (I can build a simple swift program that imports and uses SwiftDriver by pointing library and include paths to the built toolchain).
In our chat, @nkcsgexi brought up a good point that we should probably be able to refactor SwiftDriver into a client library for build systems that does not itself depend on llbuild. We can do this refactoring as either a follow-up or a pre-requisite to this change.

artemcm · 2020-09-18T00:48:52Z

@swift-ci please test

artemcm · 2020-09-18T15:55:05Z

@swift-ci please test

artemcm · 2020-09-18T20:14:22Z

Note: This must be merged in tandem with: swiftlang/swift#33918

… install libSwiftDriver Previously only the x86 `swift-driver` and `swift-help` executables were copied over to the specified toolchain's `bin` directory. With this change, libSwiftDriver shared library is also packaged and installed into the toolchain, as well as libSwiftOptions. To do so, we must build SwiftDriver and all of its dependencies using CMake for arm64 and x86_64 and create universal binaries, as documented here: https://developer.apple.com/documentation/xcode/building_a_universal_macos_binary

…e SDK

…e toolchain

…ad of static It is shared by SwiftDriver and SwiftOptions and building it as a static lib causes them to have common symbols. This does mean that we have to isntall TSC libraries into the toolchain as well

…hain, until needed Code commented-out with TODOs

artemcm · 2020-09-24T16:05:40Z

@swift-ci please test

artemcm · 2020-09-24T17:02:44Z

@swift-ci please test

DougGregor

This all looks reasonable to me. Thank you!

No longer needed with swiftlang#263

artemcm · 2020-09-25T16:22:08Z

@swift-ci please test

artemcm · 2020-09-28T15:55:38Z

@swift-ci please test

artemcm · 2020-09-28T17:45:42Z

@swift-ci please test

This was referenced Sep 11, 2020

build: begin wiring up swift-argument-parser swiftlang/swift#33901

Closed

[Do Not Merge] Test change to exercise the swift-driver CI. swiftlang/swift#33759

Closed

[Build Script] Pass down a CMake path to swift-driver's build script swiftlang/swift#33918

Merged

artemcm force-pushed the install_libSwiftDriver branch from b795c7a to 66906bd Compare September 14, 2020 17:39

artemcm force-pushed the install_libSwiftDriver branch from 66906bd to 693b1ba Compare September 15, 2020 20:53

artemcm force-pushed the install_libSwiftDriver branch from 268d83c to daef6f4 Compare September 16, 2020 23:33

artemcm changed the title ~~[WIP][Build Script Helper] On macOS make install action use CMake and install libSwiftDriver~~ [Build Script Helper] On macOS make install action use CMake and install libSwiftDriver Sep 17, 2020

artemcm force-pushed the install_libSwiftDriver branch 2 times, most recently from 3467290 to 645f15d Compare September 17, 2020 18:22

artemcm marked this pull request as ready for review September 17, 2020 18:23

artemcm requested review from shahmishal and DougGregor September 17, 2020 20:19

owenv mentioned this pull request Sep 20, 2020

Extract everything depending on LLBuild into a SwiftDriverExecution library #263

Merged

artemcm added 5 commits September 24, 2020 07:56

[Build-Script-Helper] Specify SQLite library file path directly in th…

9141daa

…e SDK

[Build Script Helper] Install all of SwiftDriver dependencies into th…

b47ac18

…e toolchain

[Build-Script-Helper] Build and install TSC as a shared library inste…

39da2ca

…ad of static It is shared by SwiftDriver and SwiftOptions and building it as a static lib causes them to have common symbols. This does mean that we have to isntall TSC libraries into the toolchain as well

[Build-Script-Helper] Avoid installing argument-parser into the toolc…

959c9f7

…hain, until needed Code commented-out with TODOs

artemcm force-pushed the install_libSwiftDriver branch from 645f15d to 0a46481 Compare September 24, 2020 16:04

artemcm changed the base branch from master to main September 24, 2020 17:02

DougGregor approved these changes Sep 24, 2020

View reviewed changes

[Build Script Helper] Remove dependency on llbuild in the install action

9d43e33

No longer needed with swiftlang#263

artemcm force-pushed the install_libSwiftDriver branch from 0a46481 to 9d43e33 Compare September 25, 2020 15:03

artemcm merged commit 0402b40 into swiftlang:main Sep 29, 2020

artemcm deleted the install_libSwiftDriver branch January 20, 2021 19:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Build Script Helper] On macOS make install action use CMake and install libSwiftDriver #246

[Build Script Helper] On macOS make install action use CMake and install libSwiftDriver #246

Uh oh!

artemcm commented Sep 11, 2020 •

edited

Loading

Uh oh!

artemcm commented Sep 15, 2020

Uh oh!

nkcsgexi commented Sep 15, 2020

Uh oh!

artemcm commented Sep 15, 2020 •

edited

Loading

Uh oh!

nkcsgexi commented Sep 15, 2020

Uh oh!

artemcm commented Sep 17, 2020

Uh oh!

artemcm commented Sep 18, 2020

Uh oh!

artemcm commented Sep 18, 2020

Uh oh!

artemcm commented Sep 18, 2020

Uh oh!

artemcm commented Sep 24, 2020

Uh oh!

artemcm commented Sep 24, 2020

Uh oh!

DougGregor left a comment

Uh oh!

artemcm commented Sep 25, 2020

Uh oh!

artemcm commented Sep 28, 2020

Uh oh!

artemcm commented Sep 28, 2020

Uh oh!

Uh oh!

[Build Script Helper] On macOS make install action use CMake and install libSwiftDriver #246

[Build Script Helper] On macOS make install action use CMake and install libSwiftDriver #246

Uh oh!

Conversation

artemcm commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Executables:

Binary Modules and Shared Libraries

C Module .modulemap and header files

Misc

Uh oh!

artemcm commented Sep 15, 2020

Uh oh!

nkcsgexi commented Sep 15, 2020

Uh oh!

artemcm commented Sep 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nkcsgexi commented Sep 15, 2020

Uh oh!

artemcm commented Sep 17, 2020

Uh oh!

artemcm commented Sep 18, 2020

Uh oh!

artemcm commented Sep 18, 2020

Uh oh!

artemcm commented Sep 18, 2020

Uh oh!

artemcm commented Sep 24, 2020

Uh oh!

artemcm commented Sep 24, 2020

Uh oh!

DougGregor left a comment

Choose a reason for hiding this comment

Uh oh!

artemcm commented Sep 25, 2020

Uh oh!

artemcm commented Sep 28, 2020

Uh oh!

artemcm commented Sep 28, 2020

Uh oh!

Uh oh!

artemcm commented Sep 11, 2020 •

edited

Loading

artemcm commented Sep 15, 2020 •

edited

Loading