Generated tests don't run reliably

[dabrahams]

Description

Check out the test-generation tag from the https://github.com/val-lang/val project, follow the setup instructions if you haven't already, and do swift -vv test. You can clearly see a test file (ValFileTests.swift) being generated, and then being compiled into the EndToEndTests test target. However, the generated XCTestCase is never run. If you open Package.swift in Xcode and run the tests there, however, they run.

There's something flaky going on here, because:

1. I've tried and failed to reproduce it in the small: https://github.com/dabrahams/TestGeneration works exactly as expected.
2. I could swear I've seen this work as expected directly in the Val project (verified; see comment below).
3. There are other weird effects, e.g. changing the source of the test generator plugin doesn't cause it to be re-run.

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.8.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.75.2 Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100)
Target: arm64-apple-macosx13.0
Darwin DaveA-MBP14.luannocracy.com 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

[tomerd]

cc @neonichu @abertelrud

[dabrahams]

I'm really tearing my hair out over this, so any help you can offer would be much appreciated! I have, from time to time, gotten my system into a state where these tests do run, even with no code changes to the project.

It appears to have something to do with state accumulated in the .build directory, but I know nothing more.

[dabrahams]

With the nightly snapshot, I get this; not sure if it's any help:

xcrun --toolchain swift swift test --filter _val
warning: 'val': found 8 file(s) which are unhandled; explicitly declare them as resources or exclude from the target
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Pragma.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Subscript.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Monomorphize.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Factorial.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Existential.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Lambda.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Destroy.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/LocalInout.val
Building for debugging...
Build complete! (0.43s)
[0/1] Planning build
[1/1] Compiling plugin Format Source Code
[2/2] Compiling plugin GenerateManualPlugin
[3/3] Compiling plugin Lint Source Code
[4/4] Compiling plugin TestGeneratorPlugin
Building for debugging...
[4/8] Generating ValFileTests.swift
[6/8] Compiling EndToEndTests ValFileTests.swift
[7/8] Emitting module EndToEndTests
ld: warning: dylib (/opt/homebrew/Cellar/llvm/16.0.3/lib/libLLVM-16.dylib) was built for newer macOS version (13.0) than being linked (12.0)
ld: warning: dylib (/opt/homebrew/Cellar/llvm/16.0.3/lib/libunwind.dylib) was built for newer macOS version (13.0) than being linked (12.0)
ld: warning: dylib (/opt/homebrew/Cellar/llvm/16.0.3/lib/libunwind.dylib) was built for newer macOS version (13.0) than being linked (12.0)
[7/8] Linking ValPackageTests
Build complete! (6.88s)
error: signalled(6): /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/dave/src/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest /var/folders/wg/wvvfbgl96f727hzqdtpfrrl80000gn/T/TemporaryFile.u89xwy output:
    dyld[99443]: Library not loaded: @rpath/XCTest.framework/Versions/A/XCTest
      Referenced from: <18FA1015-E205-3FCC-AF71-9555A3EB0095> /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper
      Reason: tried: '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/lib/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/lib/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache)

[dabrahams]

Further report: can't reproduce the problem on my work machine (MacBook Pro M1 Max), but it happens all the time on my home laptop (MacBook Pro M1). 🤔

[dabrahams]

This is amazing. SPM is unhappy with this package in a directory named val unless its package name is also "val". This change makes the problem go away:

diff --git a/Package.swift b/Package.swift
index c92ba74f..1846866b 100644
--- a/Package.swift
+++ b/Package.swift
@@ -7,7 +7,7 @@ let allTargetsSwiftSettings: [SwiftSetting] = [
 ]
 
 let package = Package(
-  name: "Val",
+  name: "val",
 
   platforms: [
     .macOS(.v12)

You'd think it would be something like, the directory name needs to match, case-sensitively, but in fact that's not it. I can make the small TestGeneration project fail by renaming its directory val/, but not testgeneration/ or TestGeneration1/, or Val/. On the other hand, renaming the package "val" doesn't fix TestGeneration once its directory has been named val/.

[dabrahams]

NOTE: for reproducibility, all invocations of swift should be prefixed with a swift package clean. Like I said, I've seen it get into a state that appears to be working, but that's gone after I clean.

[dabrahams]

Final note: TestGeneration has now been minimized so that there's nothing inessential. It runs two tests… unless I check it out into a directory named ~/src/val/ on one particular laptop (I've tried this on 4 different macs), in which case it runs zero tests. Checked out into /tmp/val or ~/Downloads/val on that machine, or anywhere else I've tried it, it runs two tests.

[dabrahams]

FWIW, the change cited here no longer seems to suppress the problem. The only answer I currently have is to move the project to a different directory.

[dabrahams]

Moving it to a directory named ~/src/val.0 doesn't suppress the problem, but ~/src/val.1 does.

[dabrahams]

Now the spm-bug-wrinkle tag in the val repository is a state where this command will run the generated tests twice, as it should:

swift package clean && swift test --filter _val && echo '*****************' && swift test --filter _val

…unless the code is in ~/src/val.1 (but not ~/src/val!), in which case, the generated tests only run the second time swift test is invoked by that command.

Is it possible that the determination of which tests need to run is happening concurrently with the test generation, without synchronization?

[dabrahams]

Bump. This bug is also reproduced by @kyouko-taiga and is a serious problem for our project. We are willing to take extraordinary measures to help debug it (e.g. run SPM under the debugger somehow) but we need to know what information you need and we need instructions to get that info.