Package loading is slow for large packages

[adam-fowler]

Description

Package loading seems unnecessarily slow for larger packages. This can be seen when running swift package describe or during a swift package resolve. The package I am seeing this most often in is https://github.com/soto-project/soto. Yes it is a large package with over 300 targets, but it can take over 15 seconds to run swift package describe on it. This seems like an incredibly long time to load the Package.swift and list all the files in each target.

A resolve can spend a similar amount of time computing the version for Soto.

Expected behavior

I would expect package loading to be faster than this

Actual behavior

No response

Steps to reproduce

1. git clone https://github.com/soto-project/soto
2. cd soto
3. swift package describe

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.75.2 Apple Swift version 5.8.1 (swiftlang-5.8.0.124.5 clang-1403.0.22.11.100)
Target: arm64-apple-macosx13.0
Darwin Adams-MBP-M1-Max.local 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000 arm64

[MaxDesiatov]

Tracked internally as rdar://113207176

[neonichu]

swift package describe is about loading the entire package graph, so it’s not limited to what is in the package itself but the entire graph.

[neonichu]

The package has 12 dependencies, that's really not that much, so ~15s seems like a lot.

[MaxDesiatov]

This is likely to be resolved by #7724, #7672, and #7452. Due to current overabundant use of semaphores waiting for callbacks to complete we've seen a variety of issues caused by this, including thread explosion, and sometimes even deadlocks in package loading and resolution.

The PRs are currently blocked due to SwiftPM plugins IPC not supporting backpressure, which will be exacerbated with a transition to async with messages possibly arriving out-of-order and other possible reentrancy issues. We need to lay some groundwork there first, which will allow us to preserve correctness in the Swift Concurrency transition.