Revert "Revert "Define commands as asynchronous and use Task for preview cancellation ""

[d-ronnqvist]

Reverts #1058

Testing

To reproduce the crash caused by rdar://137774949 you need to run the tests in a release configuration with a recent Swift compiler. For me it reproduces with swiftlang-6.1.0.1.51 but it's possible that you can go further back than that.

• Checkout a9f1c0b
• Run swift test --parallel --configuration release --filter SwiftDocCUtilitiesTests.PreviewActionIntegrationTest
  • Confirm that it reproduces the crash. If it doesn't the the steps below won't verify anything.
• Checkout 2eafe4b
• Run for i in {1..10}; do swift test --parallel --configuration release --filter SwiftDocCUtilitiesTests.PreviewActionIntegrationTests; done
  • The tests should repeatedly pass

[d-ronnqvist]

@swift-ci please test

[anferbui]

Why was it reverted, and what has changed that it's ok to bring it back now?

[d-ronnqvist]

This was reverted because it encountered a miscompile with the latest Swift compiler that resulted in a crash while running the tests which prevented a new Swift toolchain from being successfully built. (rdar://137774949)

I opened this revert and ran the CI tests because I hoped that a new toolchain would be available that reproduces the crash so that I could workaround the miscompilation but it looks like the latest main toolchain is still from October 8 so I have to wait a bit longer unless I want to build my own toolchain from source to reproduce this.

[d-ronnqvist]

Actually. I missed that this crash only reproduces in release builds. I can reproduce it with the existing swiftlang-6.1.0.1.51 version so I can work on fixing it now.

[d-ronnqvist]

I think that this successfully works around the miscompilation. I added testing steps to the PR description to verify that it works.

[d-ronnqvist]

I can reproduce the miscompilation 10/10 times on a9f1c0b (without the workaround) but 0/10 times on 2eafe4b (with the workaround) so I believe that that is fully fixed.

However, about 1/100 times on 2eafe4b I encounter a different crash in LocalFileSystemDataProvider.bundles(options:). Since that code is no longer used in #1059 I repeated the same tests there.

After running the tests in release mode 6632 times (over night) with #1059 I've encountered 0 test crashes but I hit 3 unexpected unexpected test failures due to flakiness in testCancelsConversion (at test run 3200, 5019, and 5733).

My feeling is that a ~1/100 flakiness is too frequent but a ~1/2200 flakiness is manageable. Because of this, I plan on disabling that test in this PR and reenable it in #1059. This also gives some extra time to continue to investigate the ~1/2200 flakiness of that test to see if we can get rid of it completely.

Note; I haven't had time to gather 6000 test runs on main yet because it requires running over night so I don't know if these tests have a low amount of flakiness on main as well.

[d-ronnqvist]

@swift-ci please test

[shahmishal]

Thank you for looking into the issue, would you be open to doing a cross pr testing to verify it's resolved?

[d-ronnqvist]

Yes. How do I do that?

[shahmishal]

Started the build here: swiftlang/swift#73834

[d-ronnqvist]

That seems to be crashing (or failing an assertion) with the latest compiler instead:

overlapping children
child 1:
(decl_implicit version=0 decl=filterLevel src_range=[/home/build-user/swift-docc/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticEngine.swift:32:30 - line:32:41])
child 2:
(decl_implicit version=0 decl=filterLevel src_range=[/home/build-user/swift-docc/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticEngine.swift:32:30 - line:32:41])
parent:
(decl_implicit version=0 decl=filterLevel src_range=[/home/build-user/swift-docc/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticEngine.swift:30:5 - line:36:6]
  (decl_implicit version=0 decl=filterLevel src_range=[/home/build-user/swift-docc/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticEngine.swift:32:30 - line:32:41]  )
  (decl_implicit version=0 decl=filterLevel src_range=[/home/build-user/swift-docc/Sources/SwiftDocC/Infrastructure/Diagnostics/DiagnosticEngine.swift:32:30 - line:32:41]  ))
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the crash backtrace.
Stack dump:
0.	Program arguments: /home/build-user/swift-nightly-install/usr/bin/swift-frontend -frontend -c -filelist /tmp/TemporaryDirectory.BWxb5J/sources-1 -supplementary-output-file-map /tmp/TemporaryDirectory.BWxb5J/supplementaryOutputs-1 -target x86_64-unknown-linux-gnu -disable-objc-interop -I /home/build-user/build/buildbot_linux/unified-swiftpm-build-linux-x86_64/x86_64-unknown-linux-gnu/release/Modules -g -debug-info-format=dwarf -dwarf-version=4 -module-cache-path /home/build-user/build/buildbot_linux/unified-swiftpm-build-linux-x86_64/x86_64-unknown-linux-gnu/release/ModuleCache -swift-version 5 -O -D SWIFT_PACKAGE -empty-abi-descriptor -resource-dir /home/build-user/swift-nightly-install/usr/lib/swift -file-compilation-dir /home/build-user/swift-docc -Xcc -fmodule-map-file=/home/build-user/cmark/extensions/include/module.modulemap -Xcc -I -Xcc /home/build-user/cmark/extensions/include -Xcc -fmodule-map-file=/home/build-user/cmark/src/include/module.modulemap -Xcc -I -Xcc /home/build-user/cmark/src/include -Xcc -fmodule-map-file=/home/build-user/swift-markdown/Sources/CAtomic/include/module.modulemap -Xcc -I -Xcc /home/build-user/swift-markdown/Sources/CAtomic/include -Xcc -fmodule-map-file=/home/build-user/build/buildbot_linux/unified-swiftpm-build-linux-x86_64/x86_64-unknown-linux-gnu/release/CLMDB.build/module.modulemap -Xcc -I -Xcc /home/build-user/swift-lmdb/Sources/CLMDB/include -Xcc -fmodule-map-file=/home/build-user/swift-crypto/Sources/CCryptoBoringSSLShims/include/module.modulemap -Xcc -I -Xcc /home/build-user/swift-crypto/Sources/CCryptoBoringSSLShims/include -Xcc -fmodule-map-file=/home/build-user/swift-crypto/Sources/CCryptoBoringSSL/include/module.modulemap -Xcc -I -Xcc /home/build-user/swift-crypto/Sources/CCryptoBoringSSL/include -Xcc -fPIC -Xcc -g -Xcc -fno-omit-frame-pointer -module-name SwiftDocC -package-name swift_docc -in-process-plugin-server-path /home/build-user/swift-nightly-install/usr/lib/swift/host/libSwiftInProcPluginServer.so -plugin-path /home/build-user/swift-nightly-install/usr/lib/swift/host/plugins -plugin-path /home/build-user/swift-nightly-install/usr/local/lib/swift/host/plugins -enable-default-cmo -parse-as-library -num-threads 36 -output-filelist /tmp/TemporaryDirectory.BWxb5J/outputs-1
1.	Swift version 6.1-dev (LLVM d1d2bfa54bdb520, Swift 8b712ea2f01fba2)
2.	Compiling with effective version 5.10
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend  0x00005567fc273c27
1  swift-frontend  0x00005567fc27192e
2  swift-frontend  0x00005567fc27429f
3  libpthread.so.0 0x00007fc47f4c5420
4  libc.so.6       0x00007fc47d8b100b gsignal + 203
5  libc.so.6       0x00007fc47d890859 abort + 299
6  swift-frontend  0x00005567f73325e0
7  swift-frontend  0x00005567f73324b4
8  swift-frontend  0x00005567f733224e
9  swift-frontend  0x00005567f6fd538b
10 swift-frontend  0x00005567f7321087
11 swift-frontend  0x00005567f6d1251d
12 swift-frontend  0x00005567f57bcff9
13 swift-frontend  0x00005567f57b3bfd
14 swift-frontend  0x00005567f57b3b47
15 swift-frontend  0x00005567f5478f8e
16 swift-frontend  0x00005567f546af5e
17 swift-frontend  0x00005567f5469c90
18 swift-frontend  0x00005567f5249154
19 libc.so.6       0x00007fc47d892083 __libc_start_main + 243
20 swift-frontend  0x00005567f52481ee

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

Note; I haven't had time to gather 6000 test runs on main yet because it requires running over night so I don't know if these tests have a low amount of flakiness on main as well.

I ran the same thing repeatedly on main today and after 1972 iterations it also encountered the same unexpected failure due to flakiness in testCancelsConversion so that makes me think that it wasn't introduced in #1049.

[d-ronnqvist]

I started another toolchain build on that draft PR that you linked to and it looks like it passed.

Additionally I've been investigating the remaining ~1/2000 flakiness but since it also occurs on main I plan on addressing it in a separate PR on Monday

[sofiaromorales]

I get the following output when running in 2eafe4b187dea498f62c8a177269feecc7c1555b:

swift-docc % swift test --parallel --configuration release --filter SwiftDocCUtilitiesTests.PreviewActionIntegrationTest
Building for production...
[4/4] Write swift-version-38D753C24FBC9EA2.txt
Build complete! (0.35s)
error: Exited with unexpected signal code 5
[4/4] Testing SwiftDocCUtilitiesTests.PreviewActionIntegrationTests/testHumanErrorMessageForUnavailablePort
Test Suite 'Selected tests' started at 2024-10-21 11:09:02.674.
Test Suite 'SwiftDocCPackageTests.xctest' started at 2024-10-21 11:09:02.675.
Test Suite 'PreviewActionIntegrationTests' started at 2024-10-21 11:09:02.675.
Test Case '-[SwiftDocCUtilitiesTests.PreviewActionIntegrationTests testCancelsConversion]' started.
Test Case '-[SwiftDocCUtilitiesTests.PreviewActionIntegrationTests testCancelsConversion]' passed (2.616 seconds).
Test Case '-[SwiftDocCUtilitiesTests.PreviewActionIntegrationTests testCancelsConversion]' started.

􀟈  Test run started.
􀄵  Testing Library Version: 103 (arm64e-apple-macos14.0)
􁁛  Test run with 0 tests passed after 0.001 seconds.

CC: @d-ronnqvist

[d-ronnqvist]

I get the following output when running in 2eafe4b187dea498f62c8a177269feecc7c1555b:

swift-docc % swift test --parallel --configuration release --filter SwiftDocCUtilitiesTests.PreviewActionIntegrationTest
Building for production...
[4/4] Write swift-version-38D753C24FBC9EA2.txt
Build complete! (0.35s)
error: Exited with unexpected signal code 5
[4/4] Testing SwiftDocCUtilitiesTests.PreviewActionIntegrationTests/testHumanErrorMessageForUnavailablePort
Test Suite 'Selected tests' started at 2024-10-21 11:09:02.674.
Test Suite 'SwiftDocCPackageTests.xctest' started at 2024-10-21 11:09:02.675.
Test Suite 'PreviewActionIntegrationTests' started at 2024-10-21 11:09:02.675.
Test Case '-[SwiftDocCUtilitiesTests.PreviewActionIntegrationTests testCancelsConversion]' started.
Test Case '-[SwiftDocCUtilitiesTests.PreviewActionIntegrationTests testCancelsConversion]' passed (2.616 seconds).
Test Case '-[SwiftDocCUtilitiesTests.PreviewActionIntegrationTests testCancelsConversion]' started.

􀟈  Test run started.
􀄵  Testing Library Version: 103 (arm64e-apple-macos14.0)
􁁛  Test run with 0 tests passed after 0.001 seconds.

CC: @d-ronnqvist

I'm assuming that you this the same unreliable crash that I did which is the reason why 79488db skips that test until the crashing code is no longer used in #1059:

However, about 1/100 times on 2eafe4b I encounter a different crash in LocalFileSystemDataProvider.bundles(options:). Since that code is no longer used in #1059 I repeated the same tests there.

After running the tests in release mode 6632 times (over night) with #1059 I've encountered 0 test crashes but I hit 3 unexpected unexpected test failures due to flakiness in testCancelsConversion (at test run 3200, 5019, and 5733).

My feeling is that a ~1/100 flakiness is too frequent but a ~1/2200 flakiness is manageable. Because of this, I plan on disabling that test in this PR and reenable it in #1059. This also gives some extra time to continue to investigate the ~1/2200 flakiness of that test to see if we can get rid of it completely.

Note; I haven't had time to gather 6000 test runs on main yet because it requires running over night so I don't know if these tests have a low amount of flakiness on main as well.