`swift test --parallel` seems to fail when used in GitHub actions

[dabrahams]

Description

Here's an example. It appears to be related to the size of the project / testing time; for example, this test worked fine.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

Fork the Val project, enable actions, and recreate the PR that demonstrates the failure.

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.8.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.75.2 Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100)
Target: arm64-apple-macosx13.0
Darwin DaveA-MBP14.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

[neonichu]

error: signalled(6): /Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/runner/work/val/val/.build/x86_64-apple-macosx/debug/ValPackageTests.xctest /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/TemporaryFile.YFQ7vO output:
[3474](https://github.com/val-lang/val/actions/runs/4931719079/jobs/8814040640?pr=606#step:10:3475)
    dyld[11413]: Library not loaded: '@rpath/XCTest.framework/Versions/A/XCTest'
[3475](https://github.com/val-lang/val/actions/runs/4931719079/jobs/8814040640?pr=606#step:10:3476)
      Referenced from: '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/swiftpm-xctest-helper'
[3476](https://github.com/val-lang/val/actions/runs/4931719079/jobs/8814040640?pr=606#step:10:3477)
      Reason: tried: '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../lib/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../../../lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../lib/swift-5.5/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../lib/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../../../lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../lib/swift-5.5/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/pm/../lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file)

[neonichu]

❯ otool -L usr/libexec/swift/pm/swiftpm-xctest-helper
usr/libexec/swift/pm/swiftpm-xctest-helper (architecture x86_64):
	@rpath/XCTest.framework/Versions/A/XCTest (compatibility version 0.0.0, current version 22132.0.0)
[...]

❯ otool -l usr/libexec/swift/pm/swiftpm-xctest-helper|grep -A 2 LC_RPATH
          cmd LC_RPATH
      cmdsize 72
         path @executable_path/../../../../../../../../SharedFrameworks (offset 12)
--
          cmd LC_RPATH
      cmdsize 72
         path @executable_path/../../../../../../../../SharedFrameworks (offset 12)

SharedFrameworks does not contain XCTest, so we must be setting DYLD_FRAMEWORK_PATH here.

[neonichu]

Looks like we indeed do, but we must be getting it wrong somehow in specific cases?

https://github.com/apple/swift-package-manager/blob/main/Sources/Commands/Utilities/TestingSupport.swift#L92

[neonichu]

Actually, how is this working at all? The path /Users/runner/hostedtoolcache/swift-macOS/5.8/x64/usr/libexec/swift/ suggests to me that a toolchain or CLT is being used, neither of these contain XCTest.framework, so they don't support testing at all. Only Xcode supports testing on macOS.

[neonichu]

It seems like they do install Xcode but use the swift from a toolchain which should work, presuming xcode-select or DEVELOPER_DIR is pointing to the Xcode installation and not CLT (which they also install).

[dabrahams]

Looks like we indeed do, but we must be getting it wrong somehow in specific cases?

https://github.com/apple/swift-package-manager/blob/main/Sources/Commands/Utilities/TestingSupport.swift#L92

Note that line is in an #if os(macOS) block, and tests fail on linux too, where there's no explicit setting of these environment variables.

[grynspan]

Perhaps GitHub bottlenecks process spawning?

[MahdiBM]

FWIW swift test --parallel was working fine in a 200K project with ~800 tests, (mostly non-trivial tests that do a decent amount of work like calling DB) in the short period where I had it enabled on a default GitHub Actions machine.

I think I tried setting the number of workers to 4/8 (possibly even 16) and all worked.

[grynspan]

I would expect the underlying problem to be very sensitive to timing, so what Dave was seeing might not apply to your situation.