Extreme compilation time penalty for large OpenAPI-generated file on Linux

[gwynne]

Description
In a project containing a 15,400-line source file generated by the swift-openapi-generator package, a release build takes ~3m 10s on macOS, but performing an identical build in Docker on the same machine (with Docker configured to be able to use all available CPUs and RAM) using the swiftlang/swift:nightly-5.9-jammy image (see below for version specifics) takes ~16m 30s, a penalty of almost 5x. The difference is even more severe using a swift:5.8-amazonlinux2 Intel image (almost 22 minutes). top and ps show that this is not a linker-related issue; a swift-frontend instance is the long-running process. The process spends practically all of its runtime maxing out a single core, with all other cores sitting idle, and uses less than 10% of available RAM; there is no issue of resource exhaustion.

Steps to reproduce

1. git clone https://github.com/vapor/penny-bot.git && cd penny-bot
2. git checkout 2d46427d30d9c3868a50bd6b586fe73db0967c5b
3. swift build -c release --target GHHooksLambda

Expected behavior
The compile time on Linux should be reasonable.

Environment

• macOS Ventura 13.4.1 (22F82), Xcode 15.0 beta 3 (15A5195k), swift-driver version: 1.85 Apple Swift version 5.9 (swiftlang-5.9.0.120.7 clang-1500.0.34.3)
• Docker Desktop for Mac 4.21.1 (114176), image swiftlang/swift:nightly-5.9-jammy, tag swift-5.9-DEVELOPMENT-SNAPSHOT-2023-07-09-a, Swift version 5.9-dev (LLVM edfa23b742ff932, Swift 4895334e52a9f08)

[tomerd]

cc @MaxDesiatov @al45tair @artemcm

[al45tair]

Seems pretty reproducible here (Mac finished in a bit over 3m; Linux is still going…). I'll see if I can work out where the time is going.

[al45tair]

I don't have my Linux VM configured with the same exact number of CPUs as the base machine, as the base system has hyper threading, but it took 22m, which is 7x the macOS figure…

[al45tair]

OK, we think this problem is happening in the LLVM optimizer. As a workaround, maybe try adding -Xllvm -vectorize-slp=false to the arguments handed to the Swift compiler.

[al45tair]

I've raised rdar://112463849 internally to track the underlying problem.

[gwynne]

@al45tair Thanks for the update! I can confirm that adding -Xllvm -vectorize-slp=false reduced the total build time on Linux (5.9-jammy nightly, tag swift-5.9-DEVELOPMENT-SNAPSHOT-2023-07-10-a) down to a touch under 6m, or about a 3x improvement.

[MahdiBM]

about a 3x improvement.

Pretty much the same on GitHub CI / Amazon Linux 2 too. About 34/5/6 -> 12/13 minutes. 👌