SourceKit-LSP appears to interfere with SwiftPM build plugins

[adam-fowler]

It appears that SourceKit-LSP is interfering in the build process when building a project that includes a build plugin. While running VS Code on a package that include a build plugin the linker fails to find the generated code. Sometimes subsequent builds are successful but not always. The same project compiles fine if the sourcekit-lsp process is killed.

I used VSCode with Swift extension installed to reproduce this issue. I have a sample project which you can use to reproduce.
Steps to reproduce:

1. git clone https://github.com/adam-fowler/soto-codegenerator-plugin-test
2. cd soto-codegenerator-plugin-test
3. code .
4. wait for project to load in VS Code
5. swift build (either in VS Code or from the command line)

This will produce the following link errors

Undefined symbols for architecture arm64:
  "_$s7SotoSNS0B0V0A4Core10AWSServiceAAMc", referenced from:
      _$s7SotoSNS0B0VAC0A4Core10AWSServiceAAWl in SNSTest.swift.o
  "_$s7SotoSNS0B0V11createTopic_6logger2onAC06CreateD8ResponseVAC0gD5InputV_7Logging6LoggerV7NIOCore9EventLoop_pSgtYaKF", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY0_ in SNSTest.swift.o
     (maybe you meant: _$s7SotoSNS0B0V11createTopic_6logger2onAC06CreateD8ResponseVAC0gD5InputV_7Logging6LoggerV7NIOCore9EventLoop_pSgtYaKFfA0_)
  "_$s7SotoSNS0B0V11createTopic_6logger2onAC06CreateD8ResponseVAC0gD5InputV_7Logging6LoggerV7NIOCore9EventLoop_pSgtYaKFTu", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY0_ in SNSTest.swift.o
  "_$s7SotoSNS0B0V11deleteTopic_6logger2onyAC06DeleteD5InputV_7Logging6LoggerV7NIOCore9EventLoop_pSgtYaKF", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY2_ in SNSTest.swift.o
     (maybe you meant: _$s7SotoSNS0B0V11deleteTopic_6logger2onyAC06DeleteD5InputV_7Logging6LoggerV7NIOCore9EventLoop_pSgtYaKFfA0_)
  "_$s7SotoSNS0B0V11deleteTopic_6logger2onyAC06DeleteD5InputV_7Logging6LoggerV7NIOCore9EventLoop_pSgtYaKFTu", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY2_ in SNSTest.swift.o
  "_$s7SotoSNS0B0V16CreateTopicInputV10attributes20dataProtectionPolicy4name4tagsAESDyS2SGSg_SSSgSSSayAC3TagVGSgtcfC", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY0_ in SNSTest.swift.o
  "_$s7SotoSNS0B0V16DeleteTopicInputV8topicArnAESS_tcfC", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY2_ in SNSTest.swift.o
  "_$s7SotoSNS0B0V6client6region9partition8endpoint7timeout19byteBufferAllocator7optionsAC0A4Core9AWSClientC_AK6RegionVSgAK12AWSPartitionVSSSg7NIOCore10TimeAmountVSgAT04ByteiJ0VAK16AWSServiceConfigC7OptionsVtcfC", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY0_ in SNSTest.swift.o
  "_$s7SotoSNS0B0VN", referenced from:
      _$s7SNSTest3RunV4mainyyYaKFZTY0_ in SNSTest.swift.o
      _$s7SotoSNS0B0VAC0A4Core10AWSServiceAAWl in SNSTest.swift.o
ld: symbol(s) not found for architecture arm64
```_

[ahoppen]

rdar://111918777

[adam-fowler]

Forgot to say I saw this using Swift 5.7.1 on macOS, but have also seen it using latest trunk build on Linux

[ahoppen]

@adam-fowler Are you still able to reproduce this? I tried reproducing it with swift-DEVELOPMENT-SNAPSHOT-2023-06-07-a.xctoolchain and couldn’t.

Also, in case you remember: Did you always see the issue or only if you performed certain actions in VSCode or at a certain point during package opening in VSCode, i.e. did the issue only appear right after opening it and resolved itself after a while?

[adam-fowler]

This appears to work ok with the version of swift that comes with the Xcode beta so I'll close for now. It was a little temperamental so I may re-open later on.

[adam-fowler]

Just managed to re-produce this with swift-5.9-DEVELOPMENT-SNAPSHOT-2023-07-06-a

[ahoppen]

OK, here’s what I found:

After opening the package, the VSCode plugin seems to be building all plugins. When manually invoking a build during that time, you get the error about the database.

To reproduce this:

• Delete .build
• Opening soto-codegenerator-plugin-test in VSCode
• Open the “Swift” output view
• cd .build
• while true; do ls; sleep 1; done
• Notice that after ~1-2 minutes the plugin folder appears and the Swift output view will show messages about compiling the plugins

If I wait for the initial plugin build from the VSCode extension to finish and only perform the package build afterwards, I haven’t seen any issues. So I’m thinking that this might be an issue of the VSCode extension and not a sourcekit-lsp issue after all. AFAICT sourcekit-lsp never triggers a build of the plugins that could interfere with the build, at least a breakpoint in BuildOperation.compilePlugins is never hit.

And it appears like the plugins are being built when executing swift test list --skip-build. I would say it’s a SwiftPM bug that SwiftPM builds the plugins even though --skip-build is passed. I filed swiftlang/swift-package-manager#6683 for the SwiftPM issue.

[ahoppen]

@adam-fowler To make sure there isn’t another issue lurking somewhere: Do you think it would be possible for you to try if you still see the issue if you disable the swift-test list --skip-build invocation?

[adam-fowler]

I removed the swift test list --skip-build call and it still happens. The easiest way to get it to stop is disable sourcekit-lsp. There is a setting in vscode to disable sourcekit-lsp.

[ahoppen]

I’m now able to reproduce the issue reliably as well. Mostly for my own memory, here’s what I found out.

Steps to reproduce

These steps reliably reproduced the issue for me. There might be a smaller reproducer but it’s been sufficient for me.

Preparation

1. Clone http://github.com/swift-server/vscode-swift at 2719d235
2. Apply this diff to disable the swift test list --skip-build call

diff --git a/src/TestExplorer/TestExplorer.ts b/src/TestExplorer/TestExplorer.ts
index b7efc0a..bc710ab 100644
--- a/src/TestExplorer/TestExplorer.ts
+++ b/src/TestExplorer/TestExplorer.ts
@@ -121,6 +121,7 @@ export class TestExplorer {
      * Uses `swift test --list-tests` to get the list of tests
      */
     async discoverTestsInWorkspace() {
+        return;
         try {
             const toolchain = this.folderContext.workspaceContext.toolchain;
             // get build options before build is run so we can be sure they aren't changed

3. Clone https://github.com/adam-fowler/soto-codegenerator-plugin-test
4. Open vscode-swift in Visual Studio Code

To reproduce the issue

1. Run swift package reset in soto-codegenerator-plugin-test from an external terminal
2. Debug the VSCode Swift extension (Run -> Start Debugging)
3. Open SNSTests.swift and make sure we get an error about not being able to load SotoCore to ensure SourceKit-LSP is providing semantic functionality
4. Wait for package loading to finish
5. Run /Library/Developer/Toolchains/swift-5.9-DEVELOPMENT-SNAPSHOT-2023-07-06-a.xctoolchain/usr/bin/swift build --build-tests -v from an external terminal

Finding 1: Only reproduces with sourcekit-lsp from the toolchain

If I build sourcekit-lsp locally (both in debug and release, and both when building using Xcode and build-script-helper.py) and set the sourcekit-lsp path in the VSCode plugin, the issue no longer reproduces. It does still reproduce when copying sourcekit-lsp from the toolchain to DerivedData, so it’s not related to the path at which sourcekit-lsp is located.

Finding 2: LinkFileList is missing object files of generated sources

When diffing the failed linker invocation of a successful build (with SourceKit-LSP not running) and a failed build (with SourceKit-LSP running), the only difference is that the failed invocation does not contain the following object files of generated sources.

/private/tmp/soto-codegenerator-plugin-test/.build/arm64-apple-macosx/debug/plugins/outputs/soto-codegenerator-plugin-test/SotoSNS/SotoCodeGeneratorPlugin/GeneratedSources/sns_api+async.swift.o
/private/tmp/soto-codegenerator-plugin-test/.build/arm64-apple-macosx/debug/plugins/outputs/soto-codegenerator-plugin-test/SotoSNS/SotoCodeGeneratorPlugin/GeneratedSources/sns_api.swift.o

sns_api.swift.o is the files that defines s7SotoSNS0B0V0A4Core10AWSServiceAAMc (one of the missing symbols), so the linker error makes sense. This raises the question why those files aren’t present in the linker invocation and I haven’t been able to answer that question.

After the build failed, the files are present and contain the symbol in question. That’s why a subsequent build passes. I also checked whether the MD5 hash or the presence of the file changes during the build, indicating that there is a race between swift-build and sourcekit-lsp but that’s not the case, so I’m not sure what’s going on here.

Next steps

The following are things I’d like to check next:

• Once we have a new 5.9 toolchain, check if the issue still reproduces with it. Maybe [5.9] Don’t update Package.resolved from sourcekit-lsp #763 fixed the issue but I’m not super hopeful.
• Fix Source generated by build plugin is ignored #665 because it’s also about generated sources from build plugins. Maybe they share the same underlying issue.
• Figure out which methods in SwiftPM run the build plugin to double check that SourceKit-LSP is not executing it.
• Discuss all of this with @neonichu, maybe he’s go an idea.

Another option could be that this is a race within SwiftPM itself and it somehow only happens if sourcekit-lsp is watching the files on disk for changes. I don’t think that’s the issue though, because I can reproduce it 100% with SourceKit-LSP running a never without SourceKit-LSP running and I would be expecting much higher non-determinism rates. Also, on second thought I don’t think that SourceKit-LSP is actually watching for file changes.

[ahoppen]

OK, I think I believe I have a guess what the issue is, now. Opening the project in sourcekit-lsp causes module.modulemap and Objects.LinkFileList files to be created in the build directory, which shouldn’t happen. My guess is that these are created incorrectly before the build plugins run and thus don’t include the generated source files. The swift build then picks them up and doesn’t re-generate them after running the plugin and 💥. Now I need to figure out why these files are being created.

[ahoppen]

Hooray, the issue that LinkFileList is being created has already been fixed by swiftlang/swift-package-manager#6606.

I also am not able to reproduce the issue anymore using the above instructions using swift-DEVELOPMENT-SNAPSHOT-2023-07-10-a. It’s a little unfortunate that these fixes aren’t in release/5.9 but I’m closing it as fixed.