feat: add support for Swift package registries #1318

luispadron · 2024-10-30T17:02:03Z

This adds initial support for consuming packages using a Swift Package Registry.

The majority of the changes are in the new registry_swift_package rule which creates a repository to build the targets from a package downloaded via a registry.

Notably, because the identity of the package is used to derive the repository name, packages used via a registry have the following label: @swiftpkg_[org].[name], so for example @swiftpkg_swift_collections becomes @swiftpkg_apple.swift_collections if using the swift-collections package from the apple organization (as registries support multiple orgs).

The registry_swift_package rule is mostly just an updated version of #1043 (thanks @watt)

Open questions

Should the label replace . with _ in identity? E.g. @swiftpkg_apple.swift_collections vs. @swiftpkg_apple_swift_collections

Follow-up

rule_swift_package_manager already has a bootstrapping issue, if there is no Package.resolved defined already then the Bazel repositories fail to generate. The addition of registry support adds another bootstrapping issue with the registries.json file (users will need to run swift package-registry commands to create it currently). We should address both of these by making bootstrapping a new project easier.

Caveats

There is no public Swift Package Registry (that I know of), to support testing this in CI I created a fake registry over at: https://github.com/luispadron/fake-swift-package-registry which is hosted at: https://fake-spm-registry.deno.dev
- Example: https://fake-spm-registry.deno.dev/apple/swift-collections
- I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money 😅

Example

The following shows an example of a swift_binary target depending on an SPM dependency downloaded from a registry defined in registries.json.

`Package.swift`

let package = Package(
    name: "swift_package_registry_example",
    dependencies: [
        .package(id: "apple.swift-collections", exact: "1.1.3"),
    ]
)

`Package.resolved`

{
  "pins" : [
    {
      "identity" : "apple.swift-collections",
      "kind" : "registry",
      "location" : "",
      "state" : {
        "version" : "1.1.3"
      }
    }
  ],
  "version" : 2
}

`registries.json`

{
  "authentication" : {

  },
  "registries" : {
    "[default]" : {
      "supportsAvailability" : false,
      "url" : "https://fake-spm-registry.deno.dev/"
    }
  },
  "version" : 1
}

`MODULE.bazel`

swift_deps = use_extension(
    "@rules_swift_package_manager//:extensions.bzl",
    "swift_deps",
)
swift_deps.from_package(
    registries = "//:registries.json",
    resolved = "//:Package.resolved",
    swift = "//:Package.swift",
)
swift_deps.configure_swift_package(
    replace_scm_with_registry = True,
)
use_repo(
    swift_deps,
    "swift_package",
    "swiftpkg_apple.swift_collections",
)

`BUILD.bazel`

swift_binary(
    name = "swift_package_registry_example",
    srcs = ["main.swift"],
    module_name = "swift_package_registry_example",
    visibility = ["//visibility:public"],
    deps = ["@swiftpkg_apple.swift_collections//:Collections"],
)

swiftpkg/internal/bazel_repo_names.bzl

examples/swift_package_registry_example/registries.json

luispadron · 2024-11-12T16:46:16Z

Im testing in a real registry implementation now and realizing some issues, going to make some updates but the majority of the design is ready for review

brentleyjones · 2024-11-12T16:58:22Z

We should address both of these by making bootstrapping a new project easier.

💯 (as a follow-up).

brentleyjones · 2024-11-12T17:00:58Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money

Can we spin up a local webserver as part of CI instead?

cgrindel · 2024-11-12T17:24:06Z

Should the label replace . with _ in identity? E.g. @swiftpkg_apple.swift_collections vs. @swiftpkg_apple_swift_collections

I like the underscores version. However, it is not a strong preference.

cgrindel · 2024-11-12T17:27:15Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money
Can we spin up a local webserver as part of CI instead?

+1 to @brentleyjones suggestion. Again, I am not too familiar with Swift registry hosting. If it is static, can we use GitHub to host it? I presume the functionality just needs a URL.

examples/interesting_deps/MODULE.bazel

swiftpkg/internal/swift_package_tool_attrs.bzl

luispadron · 2024-11-12T18:21:43Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money

Can we spin up a local webserver as part of CI instead?

I tried this originally but local host doesn't support https and so the bazel download stuff fails because of insecure connection.

luispadron · 2024-11-12T18:22:10Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money

Can we spin up a local webserver as part of CI instead?

+1 to @brentleyjones suggestion. Again, I am not too familiar with Swift registry hosting. If it is static, can we use GitHub to host it? I presume the functionality just needs a URL.

It is not static, it needs to be a web server which follows the registry spec.

luispadron · 2024-11-12T18:56:42Z

Im testing in a real registry implementation now and realizing some issues, going to make some updates but the majority of the design is ready for review

Done! I fixed my fake registry implementation, the .zip needed to contain a single directory named scope.name where i had name-version

cgrindel · 2024-11-12T19:13:08Z

@luispadron When you are ready to merge, if you don't want to babysit the merge, you can specify @mergifyio queue in a comment. This will add it to the merge queue that we are using.

luispadron · 2024-11-12T19:54:57Z

Should the label replace . with _ in identity? E.g. @swiftpkg_apple.swift_collections vs. @swiftpkg_apple_swift_collections

I think the answer is here is: no? Thoughts? Doing the replacement would require manipulating the identity in a fair amount of the rules and im not sure it gives us much (plus the label is still different than non registry so we don't gain anything from doing the replacement)

watt

🎉

watt · 2024-11-13T02:33:46Z

swiftpkg/internal/registry_swift_package.bzl

+        url = archive_url,
+        output = output,
+        sha256 = checksum,
+        stripPrefix = "{scope}.{name}".format(scope = id.scope, name = id.name),


This prefix isn't guaranteed unfortunately; it's going to be whatever the name of the parent folder happened to be when someone ran the publish command. I worked around this by downloading to a temp directory first and then using shell commands to move the contents back to the repo root.

Oh rough yeah, I had been using <name>-<version> for the prefix in my fake registry but then our Artifactory implementation was using scope.name.

I'll update this to work like you described since it looks like we cant use strip_prefix here

watt · 2024-11-13T02:37:16Z

swiftpkg/internal/registry_swift_package.bzl

+        "{version}.zip".format(version = version),
+    )
+    repository_ctx.download_and_extract(
+        headers = _REGISTRY_ZIP_ACCEPT_HEADER,


This is the bit that requires Bazel 7.1, which was a concern previously. Not sure if that's still a concern, but may be worth documenting as a requirement for using a registry.

I can add a comment

@cgrindel whats the minimum supported Bazel version for these rules? We need to set the headers here which is only possible in Bazel 7.1

Personally I think having this specific feature labelled as requiring Bazel 7.1 is fine. If people don't use this feature, it won't break people using less than 7.1, right? If so, it's not breaking anyone and people just need to upgrade if they want to use the new feature.

I think @brentleyjones is right. Let's document that it is only available in Bazel 7.1 or later. Go for it.

watt · 2024-11-13T02:40:20Z

swiftpkg/internal/registry_swift_package.bzl

+    )
+
+    # Download the `Package.swift` file to the root of the source archive output.
+    _download_package_swift(


Is this necessary? I think it's already inside the archive isn't it?

Yeah true.. I figured this was the point of this endpoint but we should probably just use the archive content and save ourselves a download. I'll update.

watt · 2024-11-13T02:44:16Z

swiftpkg/internal/swift_package_tool_attrs.bzl

+    "use_registry_identity_for_scm": attr.bool(
+        doc = """Look up source control dependencies in the registry and use \
+their registry identity when possible to help deduplicate across the two \
+origins.
+
+NOTE: `replace_scm_with_registry = True` implies \
+`use_registry_identity_for_scm = True`.
+""",


Aren't these exclusive? The docs say you can only choose one of the scm transformation flags.

Yeah sorry the note isn't clear, it's an error to pass both flags but thats because --replace-scm-with-registry already implies --use-registry-identity-for-scm. I'll update the doc comment here to note that both cannot be set at once though

watt · 2024-11-13T02:53:22Z

swiftpkg/internal/swift_package_tool_attrs.bzl

+    "config_path": attr.string(
+        doc = "The relative path within the runfiles tree for the Swift Package Manager configuration directory.",
+        default = ".swiftpm",
+    ),


Not quite. By default, swift looks at both a shared config in ~/.swiftpm and a local project config at .swiftpm. The --config-path argument specifies a different shared config path (there is no argument to change the project config path).

Setting this to .swiftpm (the project config path) effectively means swift will only look at .swiftpm and ignore the shared path.

Setting it to any other value means swift will look at both that value and .swiftpm.

For a hermetic build I think the first (default) behavior is maybe the only sane one, which is why I was hesitant to expose this as a configurable option previously.

Not quite. By default, swift looks at both a shared config in ~/.swiftpm and a local project config at .swiftpm. The --config-path argument specifies a different shared config path (there is no argument to change the project config path).

Yes thats how I understand this to work as well, the .swiftpm in this rule isn't in the project tree though its part of the rules runfiles.

Setting this to .swiftpm (the project config path) effectively means swift will only look at .swiftpm and ignore the shared path.

Maybe the name i picked is confusing but this .swiftpm directory is going to live in the rules runfiles, it does not mean a .swiftpm directory in the repo.

The goal of the swift_package_tool is to attempt to control where SPM looks for caches, configs, etc. By setting --config-path here SPM will use a registries.json file that exists in the swift_package_tool rules runfiles tree.

watt · 2024-11-13T02:57:48Z

swiftpkg/internal/swift_package_tool_runner_template.sh

+# If registries_json is set, symlink the `.json` file to the `config_path/configuration` directory.
+if [ -n "$registries_json" ]; then
+  mkdir -p "$config_path" && ln -sf "$(readlink -f "$registries_json")" "$config_path/registries.json"
+fi


Won't this blow away a real registries.json file if one exists? I think you're also missing the configuration directory in here.

Won't this blow away a real registries.json file if one exists?

No, this is symlinking the real file into the rules runfiles so that SPM uses that one from the rules --config-path

I think you're also missing the configuration directory in here.

Seems to work without one but i'll update to follow the normal .swiftpm layout

watt · 2024-11-13T03:18:52Z

examples/swift_package_registry_example/Package.swift

+let package = Package(
+    name: "swift_package_registry_example",
+    dependencies: [
+        .package(id: "apple.swift-collections", exact: "1.1.3"),


I ran into problems at first with --replace-scm-with-registry, because Package.resolved will contain a scoped ID like apple.swift-collections but every Package.swift will still reference the unscoped swift-collections.

To support that, I made changes to swiftpkg/internal/deps_indexes.bzl to remove the scope in _new_product_index_key. Not sure that's the ideal fix, but it worked.

Anyway, that's definitely something I would suggest testing out.

To support that, I made changes to swiftpkg/internal/deps_indexes.bzl to remove the scope in _new_product_index_key. Not sure that's the ideal fix, but it worked.

Ooh could you elaborate a bit here, is this an issue with swift package resolve, etc or with the way the rules generate something. I dont actually know whats required in a normal SPM project to go from SCM -> Registry, do folks need to delete the Package.resolved and re-resolve to replace the SCM identities?

brentleyjones · 2024-11-13T19:54:31Z

I think the answer is here is: no?

Yeah, let's not do the replacement.

luispadron force-pushed the luis/add-swift-package-registry-support branch 2 times, most recently from a5adfda to 6739de1 Compare October 30, 2024 17:08

brentleyjones reviewed Oct 30, 2024

View reviewed changes

swiftpkg/internal/bazel_repo_names.bzl Outdated Show resolved Hide resolved

luispadron force-pushed the luis/add-swift-package-registry-support branch 4 times, most recently from 452f05d to ad96158 Compare November 5, 2024 07:15

luispadron force-pushed the luis/add-swift-package-registry-support branch 2 times, most recently from 671c25d to f921aba Compare November 12, 2024 02:41

luispadron marked this pull request as ready for review November 12, 2024 02:49

luispadron requested review from brentleyjones, cgrindel and jpsim November 12, 2024 02:51

luispadron force-pushed the luis/add-swift-package-registry-support branch 3 times, most recently from a541b18 to 9838fcd Compare November 12, 2024 04:50

luispadron commented Nov 12, 2024

View reviewed changes

examples/swift_package_registry_example/registries.json Outdated Show resolved Hide resolved

luispadron force-pushed the luis/add-swift-package-registry-support branch 4 times, most recently from 3d7452d to 01abd62 Compare November 12, 2024 16:45

cgrindel reviewed Nov 12, 2024

View reviewed changes

examples/interesting_deps/MODULE.bazel Outdated Show resolved Hide resolved

swiftpkg/internal/swift_package_tool_attrs.bzl Outdated Show resolved Hide resolved

feat: add support for Swift package registries

2a46f9e

luispadron force-pushed the luis/add-swift-package-registry-support branch from 01abd62 to 2a46f9e Compare November 12, 2024 18:17

cgrindel approved these changes Nov 12, 2024

View reviewed changes

watt approved these changes Nov 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for Swift package registries #1318

feat: add support for Swift package registries #1318

luispadron commented Oct 30, 2024 •

edited

Loading

luispadron commented Nov 12, 2024 •

edited

Loading

brentleyjones commented Nov 12, 2024

brentleyjones commented Nov 12, 2024

cgrindel commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

watt left a comment

watt Nov 13, 2024

luispadron Nov 13, 2024

watt Nov 13, 2024

luispadron Nov 13, 2024

brentleyjones Nov 13, 2024 •

edited

Loading

cgrindel Nov 13, 2024

watt Nov 13, 2024

luispadron Nov 13, 2024

watt Nov 13, 2024

luispadron Nov 13, 2024

watt Nov 13, 2024

luispadron Nov 13, 2024

watt Nov 13, 2024

luispadron Nov 13, 2024

luispadron Nov 13, 2024

watt Nov 13, 2024

luispadron Nov 13, 2024

brentleyjones commented Nov 13, 2024

feat: add support for Swift package registries #1318

Are you sure you want to change the base?

feat: add support for Swift package registries #1318

Conversation

luispadron commented Oct 30, 2024 • edited Loading

Open questions

Follow-up

Caveats

Example

Package.swift

Package.resolved

registries.json

MODULE.bazel

BUILD.bazel

luispadron commented Nov 12, 2024 • edited Loading

brentleyjones commented Nov 12, 2024

brentleyjones commented Nov 12, 2024

cgrindel commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

watt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brentleyjones Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brentleyjones commented Nov 13, 2024

luispadron commented Oct 30, 2024 •

edited

Loading

`Package.swift`

`Package.resolved`

`registries.json`

`MODULE.bazel`

`BUILD.bazel`

luispadron commented Nov 12, 2024 •

edited

Loading

brentleyjones Nov 13, 2024 •

edited

Loading