Handle paginated registry metadata responses

[plemarquand]

Motivation:

In 4.1 List Package Releases it is stated that a server may respond with a Link header that contains a pointer to a subsequent page of results. SwiftPM is not checking for the next link in the Link header and so if a registry returns paginated results, only the first page of versions is searched when resolving. This would result in a "version not found" error when in reality the version is present in a subsequent page of results.

Modifications:

Respect the next link in the Link header by loading the next page of results and building up a list of versions, continuing until there is no next link present in the Link header of the last result.

Result:

Registry servers that serve paginated results now have all their results read.

Issue: #8215

[plemarquand]

@swift-ci test

[plemarquand]

@swift-ci test windows

[MaxDesiatov]

With the unstructured task spawned, how does this handle cancellation?

[plemarquand]

Good point, cancellation is not handled, though was it handled previously? As far as I can tell the callbackQueue typically provided is DispatchQueue.sharedConcurrent and DispatchQueue doesn't have a nice mechanism for this.

[plemarquand]

Perhaps:

@available(*, noasync, message: "Use the async alternative")
public func getPackageMetadata(
    package: PackageIdentity,
    timeout: DispatchTimeInterval? = .none,
    observabilityScope: ObservabilityScope,
    callbackQueue: DispatchQueue,
    completion: @escaping (Result<PackageMetadata, Error>) -> Void
) {
    let completion = self.makeAsync(completion, on: callbackQueue)
    _ = Task {
        do {
            let result = try await self.getPackageMetadata(
                package: package,
                timeout: timeout,
                observabilityScope: observabilityScope,
                callbackQueue: callbackQueue
            )

            guard !Task.isCancelled else {
                completion(.failure(CancellationError()))
                return
            }
            completion(.success(result))
        } catch {
            guard !Task.isCancelled else {
                completion(.failure(CancellationError()))
                return
            }
            completion(.failure(error))
        }
    }
}

[MaxDesiatov]

Task.isCancelled will always be false since only structured child tasks get cancellation propagated from parent tasks. Task is an unstructured task with no parents, the only primitives for structured concurrency that propagate cancellation are async let and withTaskGroup.

[plemarquand]

OK, that is consistent with my understanding. Seeing as the sync version of the API didn't handle cancellation before I think its reasonable to keep that behaviour.

However you're correct that the async version should handle cancellation, and I've updated the PR accordingly.

[plemarquand]

@swift-ci test

[cmcgee1024]

question: Where'd this observability message go?

[cmcgee1024]

praise: There's one more happy kitten and this is less loud.

[cmcgee1024]

thought: I wonder if there is an openapi spec for the package registries. It probably wouldn't handle pagination out of the box (an assumption). But, it might eliminate some boilerplate and make mocking with Swift datatypes instead of JSON payloads.

[jakepetroules]

@czechboy0 would know if OpenAPI specs have a way to communicate pagination.

[czechboy0]

Yes I refreshed the OpenAPI doc a few months ago, and if used for codegen, it would remove a bunch of the manual networking code here: https://github.com/swiftlang/swift-package-manager/blob/main/Documentation/PackageRegistry/registry.openapi.yaml

OpenAPI itself doesn't get as high level as providing conventions for pagination. But the generated code will give you the Link header, it's documented in the OpenAPI doc.

[czechboy0]

If you adopt Swift OpenAPI Generator, I think it'd be reasonable to file an issue on us to provide the Link header parsing out of the box, so you'd get typesafe access to the individual link values. That way you could drop most of this manual parsing code.

[cmcgee1024]

Overall this looks good to me. I've just added a few thoughts and questions.