Skip to content

Commit

Permalink
Add ability to bundle dylibs into CLI binary and load at startup
Browse files Browse the repository at this point in the history
Adding SwiftSyntax to the generator requires the dylib
`_InternalSwiftSyntaxParser` which is normally bundled as part of the
Xcode toolchain. Due to instability and crashing in SwiftSyntax for
Swift 5.1, we are using SwiftSyntax for Swift 5.2 and thus requires the
Xcode 11.4 toolchain.

See also: realm/SwiftLint#3105

Several potential solutions I reviewed:

1. Only support running with Xcode 11.4

This is unacceptable because we want the CLI to run mostly independently
of what Xcode toolchain is currently installed on the machine.

2. Provide the dylib when installing the CLI

This is a straightforward option and works out-of-the-box if the dylib
is in the same directory as the CLI (due to the order of rpath
expansion by dyld). However, now the CLI is no longer easily portable
and we'd need to create an installation method for all supported package
managers.

3. Embed the dylib into the CLI

This incurs a small complexity in the launch process but provides the
most streamlined experience for developers. At a high level it involves
embedding the dylib into the source and at runtime outputting it into a
directory under a custom rpath.

Once all dependent dylibs exist, it's not possible to just use runtime
loading calls `dlopen` / `dlsym` unless we fork SwiftSyntax to pull the
symbols dynamically.

It's also not possible to call into dyld APIs to replace the loaded but
unbound symbol references after the program has launched (`__dyld_start`
has already completed).

https://opensource.apple.com/source/dyld

The best approach here seems to be to relaunch the current process as a
subprocess with the same args and env.
  • Loading branch information
andrewchang-bird committed Apr 25, 2020
1 parent 4514aa2 commit d073e29
Show file tree
Hide file tree
Showing 17 changed files with 594 additions and 24 deletions.
58 changes: 58 additions & 0 deletions .github/workflows/build-framework-cli.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,34 @@ jobs:
name: Run on latest macOS
runs-on: macOS-latest

steps:
- uses: actions/checkout@v1
- name: Print Debug Info
run: make print-debug-info
- name: Set Up Project
run: make setup-project
- name: Clean
run: make clean
- name: Build
run: make build
- name: Install
run: 'export PREFIX=$(pwd) && make install'
- name: Set Up Target
run: |
./bin/mockingbird install \
--target MockingbirdTests \
--source MockingbirdTestsHost \
--loglevel verbose \
--verbose
- name: Test
run: make clean-test
- name: Cached Test
run: make test

build-xcode-11_4_0:
name: Xcode 11.4.0 toolchain
runs-on: macOS-latest

steps:
- uses: actions/checkout@v1
- name: Set Up Environment
Expand All @@ -32,3 +60,33 @@ jobs:
run: make clean-test
- name: Cached Test
run: make test

build-xcode-11_3_1:
name: Xcode 11.3.1 toolchain
runs-on: macOS-latest

steps:
- uses: actions/checkout@v1
- name: Set Up Environment
run: sudo xcode-select -s /Applications/Xcode_11.3.1.app/Contents/Developer
- name: Print Debug Info
run: make print-debug-info
- name: Set Up Project
run: make setup-project
- name: Clean
run: make clean
- name: Build
run: make build
- name: Install
run: 'export PREFIX=$(pwd) && make install'
- name: Set Up Target
run: |
./bin/mockingbird install \
--target MockingbirdTests \
--source MockingbirdTestsHost \
--loglevel verbose \
--verbose
- name: Test
run: make clean-test
- name: Cached Test
run: make test
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ mockingbird
## Caches
*.xcodeproj/MockingbirdCache/*.lock

## Embedded resources
MockingbirdCli/Libraries/*.generated.swift

## Various settings
*.pbxuser
!default.pbxuser
Expand Down
56 changes: 45 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@ BUILD_TOOL?=xcodebuild
TEMPORARY_FOLDER=$(TEMPORARY_FOLDER_ROOT)/Mockingbird.make.dst
TEMPORARY_INSTALLER_FOLDER=$(TEMPORARY_FOLDER)/install
XCODEBUILD_DERIVED_DATA=$(TEMPORARY_FOLDER)/xcodebuild/DerivedData/MockingbirdFramework
XCODE_PATH=$(shell xcode-select --print-path)

SIMULATOR_NAME=iphone11-mockingbird
SIMULATOR_DEVICE_TYPE=com.apple.CoreSimulator.SimDeviceType.iPhone-11
SIMULATOR_RUNTIME=com.apple.CoreSimulator.SimRuntime.iOS-13-3

SWIFT_BUILD_FLAGS=--configuration release
SWIFT_BUILD_FLAGS=--configuration release -Xlinker -weak-l_InternalSwiftSyntaxParser
XCODEBUILD_FLAGS=-project 'Mockingbird.xcodeproj' DSTROOT=$(TEMPORARY_FOLDER)
XCODEBUILD_MACOS_FLAGS=$(XCODEBUILD_FLAGS) -destination 'platform=OS X'
XCODEBUILD_FRAMEWORK_FLAGS=$(XCODEBUILD_FLAGS) \
Expand All @@ -32,6 +33,9 @@ EXAMPLE_CARTHAGE_XCODEBUILD_FLAGS=$(EXAMPLE_XCODEBUILD_FLAGS) \
FRAMEWORKS_FOLDER=/Library/Frameworks
BINARIES_FOLDER=$(PREFIX)/bin

DEFAULT_XCODE_RPATH=$(XCODE_PATH)/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx
MOCKINGBIRD_RPATH=/var/tmp/mockingbird/$(VERSION_STRING)/libs

PKG_BUNDLE_IDENTIFIER=co.bird.mockingbird
PKG_IDENTITY_NAME=3rd Party Mac Developer Installer: Bird Rides, Inc. (P2T4T6R4SL)
ZIP_IDENTITY_NAME=3rd Party Mac Developer Application: Bird Rides, Inc. (P2T4T6R4SL)
Expand Down Expand Up @@ -59,7 +63,11 @@ APPLETVSIMULATOR_FRAMEWORK_PATH=$(APPLETVSIMULATOR_FRAMEWORK_FOLDER)/$(FRAMEWORK
LICENSE_FILENAME=LICENSE
LICENSE_PATH=$(shell pwd)/$(LICENSE_FILENAME)

INSTALLABLE_FILENAMES="$(CLI_FILENAME)" "$(MACOS_FRAMEWORK_FILENAME)" "$(IPHONESIMULATOR_FRAMEWORK_FILENAME)" "$(APPLETVSIMULATOR_FRAMEWORK_FILENAME)" "$(LICENSE_FILENAME)"
INSTALLABLE_FILENAMES="$(CLI_FILENAME)" \
"$(MACOS_FRAMEWORK_FILENAME)" \
"$(IPHONESIMULATOR_FRAMEWORK_FILENAME)" \
"$(APPLETVSIMULATOR_FRAMEWORK_FILENAME)" \
"$(LICENSE_FILENAME)"

OUTPUT_PACKAGE=Mockingbird.pkg
OUTPUT_ZIP=Mockingbird.zip
Expand All @@ -81,6 +89,7 @@ ERROR_MSG=[ERROR] The downloaded Mockingbird CLI binary does not have the expect
clean \
setup-project \
save-xcschemes \
generate-embedded-dylibs \
build-cli \
build-framework-macos \
build-framework-iphonesimulator \
Expand Down Expand Up @@ -138,16 +147,41 @@ setup-project:
save-xcschemes:
cp -rf Mockingbird.xcodeproj/xcshareddata/xcschemes/*.xcscheme Xcode/XCSchemes

print-debug-info:
pwd
ls -la
swift --version
xcode-select --print-path
xcodebuild -version
xcodebuild -showsdks

build-cli:
# Generate a random number.
# This is not run initially.
GENERATE_ID = $(shell od -vAn -N2 -tu2 < /dev/urandom)

# Generate a random number, and assign it to MY_ID
# This is not run initially.
SET_ID = $(eval MY_ID=$(GENERATE_ID))

print-debug-info:
@echo "Mockingbird version: $(VERSION_STRING)"
@echo "Installation prefix: $(PREFIX)"
@echo "Temporary folder: $(TEMPORARY_FOLDER_ROOT)"
@echo "Build tool: $(BUILD_TOOL)"
$(eval XCODE_PATH_VAR = $(XCODE_PATH))
@echo "Xcode path: $(XCODE_PATH_VAR)"
@echo "Built CLI path: $(EXECUTABLE_PATH)"
$(eval CURRENT_REV = $(shell git rev-parse HEAD))
@echo "Current revision: $(CURRENT_REV)"
$(eval SWIFT_VERSION = $(shell swift --version))
@echo "Swift version: $(SWIFT_VERSION)"
$(eval XCODEBUILD_VERSION = $(shell xcodebuild -version))
@echo "Xcodebuild version: $(XCODEBUILD_VERSION)"

generate-embedded-dylibs:
Scripts/generate-resource-file.sh \
MockingbirdCli/Libraries/lib_InternalSwiftSyntaxParser.dylib \
MockingbirdCli/Libraries/SwiftSyntaxParserDylib.generated.swift \
'swiftSyntaxParserDylib'

build-cli: generate-embedded-dylibs
swift build $(SWIFT_BUILD_FLAGS) --product mockingbird
# Inject custom rpath into binary.
$(eval RPATH = $(DEFAULT_XCODE_RPATH))
install_name_tool -delete_rpath "$(RPATH)" "$(EXECUTABLE_PATH)"
install_name_tool -add_rpath "$(MOCKINGBIRD_RPATH)" "$(EXECUTABLE_PATH)"

build-framework-macos:
$(BUILD_TOOL) -scheme 'MockingbirdFramework' -configuration 'Release' -sdk macosx -arch x86_64 $(XCODEBUILD_FRAMEWORK_FLAGS)
Expand Down
81 changes: 80 additions & 1 deletion Mockingbird.xcodeproj/project.pbxproj
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@
/* Begin PBXBuildFile section */
5D5FAC84DF438200D4E44D85 /* MockingbirdTestsHostMocks.generated.swift in Sources */ = {isa = PBXBuildFile; fileRef = 93974737AF2D18DBCBCCA8F1 /* MockingbirdTestsHostMocks.generated.swift */; };
D372A4622454DFB60000E80A /* SequentialValueStubbingTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = D372A4612454DFB60000E80A /* SequentialValueStubbingTests.swift */; };
D372A44D2453CB2A0000E80A /* OutputStream+Extensions.swift in Sources */ = {isa = PBXBuildFile; fileRef = D372A44C2453CB2A0000E80A /* OutputStream+Extensions.swift */; };
D372A4592454A5DD0000E80A /* SwiftSyntaxParserDylib.generated.swift in Sources */ = {isa = PBXBuildFile; fileRef = D372A4492453C4C40000E80A /* SwiftSyntaxParserDylib.generated.swift */; };
D372A45B2454A7110000E80A /* LoadDylib.swift in Sources */ = {isa = PBXBuildFile; fileRef = D372A44E2453CCF60000E80A /* LoadDylib.swift */; };
D372A45D2454ADCB0000E80A /* Resource.swift in Sources */ = {isa = PBXBuildFile; fileRef = D372A45C2454ADCB0000E80A /* Resource.swift */; };
D3D936F22450F61400078751 /* MockingbirdTestsHost.h in Headers */ = {isa = PBXBuildFile; fileRef = OBJ_130 /* MockingbirdTestsHost.h */; settings = {ATTRIBUTES = (Public, ); }; };
D3D936F62450F85100078751 /* ParsedFile.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3D936F52450F85100078751 /* ParsedFile.swift */; };
D3D936F82451606000078751 /* ParseSingleFileOperation.swift in Sources */ = {isa = PBXBuildFile; fileRef = D3D936F72451606000078751 /* ParseSingleFileOperation.swift */; };
Expand Down Expand Up @@ -1104,6 +1108,12 @@
93974737AF2D18DBCBCCA8F1 /* MockingbirdTestsHostMocks.generated.swift */ = {isa = PBXFileReference; explicitFileType = sourcecode.swift; name = MockingbirdTestsHostMocks.generated.swift; path = MockingbirdMocks/MockingbirdTestsHostMocks.generated.swift; sourceTree = "<group>"; };
"AEXML::AEXML::Product" /* AEXML.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; path = AEXML.framework; sourceTree = BUILT_PRODUCTS_DIR; };
D372A4612454DFB60000E80A /* SequentialValueStubbingTests.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = SequentialValueStubbingTests.swift; sourceTree = "<group>"; };
D372A4422452CF280000E80A /* lib_InternalSwiftSyntaxParser.dylib */ = {isa = PBXFileReference; lastKnownFileType = "compiled.mach-o.dylib"; name = lib_InternalSwiftSyntaxParser.dylib; path = MockingbirdGenerator/Libraries/lib_InternalSwiftSyntaxParser.dylib; sourceTree = "<group>"; };
D372A4452452D9280000E80A /* lib_InternalSwiftSyntaxParser.dylib */ = {isa = PBXFileReference; lastKnownFileType = "compiled.mach-o.dylib"; path = lib_InternalSwiftSyntaxParser.dylib; sourceTree = "<group>"; };
D372A4492453C4C40000E80A /* SwiftSyntaxParserDylib.generated.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = SwiftSyntaxParserDylib.generated.swift; sourceTree = "<group>"; };
D372A44C2453CB2A0000E80A /* OutputStream+Extensions.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = "OutputStream+Extensions.swift"; sourceTree = "<group>"; };
D372A44E2453CCF60000E80A /* LoadDylib.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = LoadDylib.swift; sourceTree = "<group>"; };
D372A45C2454ADCB0000E80A /* Resource.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Resource.swift; sourceTree = "<group>"; };
D3A53F412450C59F00A1DB4B /* MockingbirdTests.xcconfig */ = {isa = PBXFileReference; lastKnownFileType = text.xcconfig; path = MockingbirdTests.xcconfig; sourceTree = "<group>"; };
D3A53F422450C59F00A1DB4B /* MockingbirdModuleTestsHost.xcconfig */ = {isa = PBXFileReference; lastKnownFileType = text.xcconfig; path = MockingbirdModuleTestsHost.xcconfig; sourceTree = "<group>"; };
D3A53F432450C59F00A1DB4B /* MockingbirdFramework.xcconfig */ = {isa = PBXFileReference; lastKnownFileType = text.xcconfig; path = MockingbirdFramework.xcconfig; sourceTree = "<group>"; };
Expand Down Expand Up @@ -1897,6 +1907,32 @@
name = "Generated Mocks";
sourceTree = "<group>";
};
D372A4412452CF280000E80A /* Frameworks */ = {
isa = PBXGroup;
children = (
D372A4422452CF280000E80A /* lib_InternalSwiftSyntaxParser.dylib */,
);
name = Frameworks;
sourceTree = "<group>";
};
D372A4442452D9280000E80A /* Libraries */ = {
isa = PBXGroup;
children = (
D372A4452452D9280000E80A /* lib_InternalSwiftSyntaxParser.dylib */,
D372A4492453C4C40000E80A /* SwiftSyntaxParserDylib.generated.swift */,
);
path = Libraries;
sourceTree = "<group>";
};
D372A45E2454ADDF0000E80A /* Launcher */ = {
isa = PBXGroup;
children = (
D372A44E2453CCF60000E80A /* LoadDylib.swift */,
D372A45C2454ADCB0000E80A /* Resource.swift */,
);
path = Launcher;
sourceTree = "<group>";
};
D3A53F402450C59F00A1DB4B /* XCConfigs */ = {
isa = PBXGroup;
children = (
Expand Down Expand Up @@ -1946,6 +1982,7 @@
OBJ_111 /* Log.swift */,
OBJ_113 /* OperationQueue+Extensions.swift */,
OBJ_112 /* OSLog+Extensions.swift */,
D372A44C2453CB2A0000E80A /* OutputStream+Extensions.swift */,
OBJ_114 /* Path+Fnmatch.swift */,
OBJ_115 /* Path+WriteUtf8Strings.swift */,
OBJ_116 /* String+Extensions.swift */,
Expand Down Expand Up @@ -2916,6 +2953,7 @@
OBJ_668 /* README.md */,
OBJ_669 /* CONTRIBUTING.md */,
OBJ_670 /* MockingbirdFramework.podspec */,
D372A4412452CF280000E80A /* Frameworks */,
);
sourceTree = "<group>";
};
Expand Down Expand Up @@ -3330,6 +3368,8 @@
isa = PBXGroup;
children = (
OBJ_9 /* Info.plist */,
D372A45E2454ADDF0000E80A /* Launcher */,
D372A4442452D9280000E80A /* Libraries */,
OBJ_10 /* Interface */,
OBJ_22 /* main.swift */,
);
Expand All @@ -3343,11 +3383,11 @@
OBJ_90 /* CheckCacheOperation.swift */,
OBJ_92 /* ExtractSourcesOperation.swift */,
OBJ_93 /* FlattenInheritanceOperation.swift */,
OBJ_94 /* SwiftFileParser.swift */,
OBJ_95 /* ParseFilesOperation.swift */,
D3D936F72451606000078751 /* ParseSingleFileOperation.swift */,
OBJ_96 /* ProcessStructuresOperation.swift */,
OBJ_97 /* ProcessTypesOperation.swift */,
OBJ_94 /* SwiftFileParser.swift */,
);
path = Operations;
sourceTree = "<group>";
Expand Down Expand Up @@ -3444,6 +3484,7 @@
isa = PBXNativeTarget;
buildConfigurationList = OBJ_779 /* Build configuration list for PBXNativeTarget "MockingbirdCli" */;
buildPhases = (
D372A45A2454A5F00000E80A /* Generate Embedded Dylibs */,
OBJ_782 /* Sources */,
OBJ_794 /* Frameworks */,
);
Expand Down Expand Up @@ -3983,6 +4024,11 @@
attributes = {
LastSwiftMigration = 9999;
LastUpgradeCheck = 1140;
TargetAttributes = {
"Mockingbird::MockingbirdGenerator" = {
LastSwiftMigration = 1140;
};
};
};
buildConfigurationList = OBJ_2 /* Build configuration list for PBXProject "Mockingbird" */;
compatibilityVersion = "Xcode 3.2";
Expand Down Expand Up @@ -4052,6 +4098,27 @@
shellPath = /bin/sh;
shellScript = "${BUILT_PRODUCTS_DIR}/MockingbirdCli generate \\\n --targets 'MockingbirdTestsHost' \\\n --outputs \"${SRCROOT}/MockingbirdMocks/MockingbirdTestsHostMocks.generated.swift\" \\\n --support \"${SRCROOT}/MockingbirdSupport\" \\\n --disable-cache \\\n --verbose\n\n# Ensure mocks are generated prior to running Compile Sources\nrm -f '/tmp/Mockingbird-718E12D6-8781-4D1E-92F1-32213EBE78AA'\n";
};
D372A45A2454A5F00000E80A /* Generate Embedded Dylibs */ = {
isa = PBXShellScriptBuildPhase;
buildActionMask = 2147483647;
files = (
);
inputFileListPaths = (
);
inputPaths = (
"$(SRCROOT)/MockingbirdCli/Libraries/lib_InternalSwiftSyntaxParser.dylib",
"",
);
name = "Generate Embedded Dylibs";
outputFileListPaths = (
);
outputPaths = (
"$(SRCROOT)/MockingbirdCli/Libraries/SwiftSyntaxParserDylib.generated.swift",
);
runOnlyForDeploymentPostprocessing = 0;
shellPath = /bin/sh;
shellScript = "\"${SRCROOT}/Scripts/generate-resource-file.sh\" \"${SCRIPT_INPUT_FILE_0}\" \"${SCRIPT_OUTPUT_FILE_0}\" 'swiftSyntaxParserDylib'\n";
};
E0C37AF4D09E1A81AA94E903 /* Clean Mockingbird Mocks */ = {
isa = PBXShellScriptBuildPhase;
buildActionMask = 2147483647;
Expand Down Expand Up @@ -4593,14 +4660,17 @@
OBJ_783 /* ArgumentParser+Extensions.swift in Sources */,
OBJ_784 /* GenerateCommand.swift in Sources */,
OBJ_785 /* InstallCommand.swift in Sources */,
D372A4592454A5DD0000E80A /* SwiftSyntaxParserDylib.generated.swift in Sources */,
OBJ_786 /* TestbedCommand.swift in Sources */,
OBJ_787 /* UninstallCommand.swift in Sources */,
OBJ_788 /* VersionCommand.swift in Sources */,
OBJ_789 /* Generator.swift in Sources */,
OBJ_790 /* Installer.swift in Sources */,
D372A45B2454A7110000E80A /* LoadDylib.swift in Sources */,
OBJ_791 /* LocalizedError+Extensions.swift in Sources */,
OBJ_792 /* Program.swift in Sources */,
OBJ_793 /* main.swift in Sources */,
D372A45D2454ADCB0000E80A /* Resource.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
Expand Down Expand Up @@ -4649,6 +4719,7 @@
OBJ_879 /* MockableTypeInitializerTemplate.swift in Sources */,
OBJ_880 /* MockableTypeTemplate.swift in Sources */,
OBJ_881 /* Template.swift in Sources */,
D372A44D2453CB2A0000E80A /* OutputStream+Extensions.swift in Sources */,
OBJ_882 /* VariableTemplate.swift in Sources */,
OBJ_883 /* GenericType.swift in Sources */,
D3D936FA245193E000078751 /* Deallocation.swift in Sources */,
Expand Down Expand Up @@ -6506,6 +6577,10 @@
);
INFOPLIST_FILE = Mockingbird.xcodeproj/MockingbirdGenerator_Info.plist;
LD_RUNPATH_SEARCH_PATHS = "$(inherited) $(TOOLCHAIN_DIR)/usr/lib/swift/macosx";
LIBRARY_SEARCH_PATHS = (
"$(inherited)",
"$(PROJECT_DIR)/MockingbirdGenerator/Libraries",
);
OTHER_CFLAGS = "$(inherited)";
OTHER_LDFLAGS = "$(inherited)";
OTHER_SWIFT_FLAGS = "$(inherited) -Xcc -fmodule-map-file=$(SRCROOT)/Mockingbird.xcodeproj/GeneratedModuleMap/_CSwiftSyntax/module.modulemap";
Expand Down Expand Up @@ -6538,6 +6613,10 @@
);
INFOPLIST_FILE = Mockingbird.xcodeproj/MockingbirdGenerator_Info.plist;
LD_RUNPATH_SEARCH_PATHS = "$(inherited) $(TOOLCHAIN_DIR)/usr/lib/swift/macosx";
LIBRARY_SEARCH_PATHS = (
"$(inherited)",
"$(PROJECT_DIR)/MockingbirdGenerator/Libraries",
);
OTHER_CFLAGS = "$(inherited)";
OTHER_LDFLAGS = "$(inherited)";
OTHER_SWIFT_FLAGS = "$(inherited) -Xcc -fmodule-map-file=$(SRCROOT)/Mockingbird.xcodeproj/GeneratedModuleMap/_CSwiftSyntax/module.modulemap";
Expand Down
Loading

0 comments on commit d073e29

Please sign in to comment.