Surprising differences in output formatting when testing --parallel or on Linux.

[dabrahams]

Description

Check out https://github.com/dabrahams/TestGeneration and run

swift test && swift test --parallel

I was very surprised by the difference in output format:

Test Case '-[TestGenerationTests.GeneratedTests test_one]' started.
Test Case '-[TestGenerationTests.GeneratedTests test_one]' passed (0.000 seconds).
Test Case '-[TestGenerationTests.GeneratedTests test_two]' started.
Test Case '-[TestGenerationTests.GeneratedTests test_two]' passed (0.000 seconds).

For the non-parallel version and

[1/2] Testing TestGenerationTests.GeneratedTests/test_two
[2/2] Testing TestGenerationTests.GeneratedTests/test_one

For the parallel one. Note the slash-vs-space in addition to the more obvious differences.

This came up as I was trying to account for this bug to make sure our CI was actually running all intended tests.

So it turns out on Linux the output without --parallel has yet another different format:

Test Case 'GeneratedTests.test_one' started at 2023-05-29 01:37:11.406
Test Case 'GeneratedTests.test_one' passed (0.0 seconds)
Test Case 'GeneratedTests.test_two' started at 2023-05-29 01:37:11.406
Test Case 'GeneratedTests.test_two' passed (0.0 seconds)

As a result I ended up with a pretty wacky regexp to validate that these tests are running. More uniformity would make that a lot simpler!

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.8.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.75.2 Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100)
Target: arm64-apple-macosx13.0
Darwin DaveA-MBP14.localdomain 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

[neonichu]

Agreed that these should be unified.

My understanding is that this is due to implementation details:

• the sequential tests show the XCTest output verbatim
• the parallel tests have their own output coming from SwiftPM (presumably to avoid mixed up output)
• Linux output might be due to open source XCTest having different output

@stmontgomery do you happen to know whether it is known that the different XCTest implementations produce different output?

[grynspan]

We're not likely going to change the formatting of console output because there are too many scrapey tools out there already. One thing we can look at is teaching XCTest to adopt the same JSON schema as is used by Swift Testing, which would give you something better than human-readable text to parse.