Feature Request: Add a rule for publishing to a crate registry

[UebelAndre]

I think it'd be hugely beneficial to have rules that could package and publish a crate built in bazel to a published registry. Is anyone working on such a thing? Or is there some other implementation I'm not aware of.

[jon-chuang]

I’d also like to see rust projects built via Bazel that can be packaged as a cargo crate and be published on crates.io. This might be somewhat outside the scope of rules_rust, but since Bazel is so powerful anyway, I figure all of the linking and packaging can be handled via Bazel.

Although, looking at how tensorflow-sys handles it, it does seem that one can handle all of the above via build.rs anyway. Maybe I should investigate that pathway first.

[illicitonion]

There's some prior art as to how the JVM world does this in rules_jvm_external's java_export rule.

A rough sketch of what I imagine this could look like in Rust:

A cargo package is a .tar.gz of a Cargo.toml file, a series of source files in a layout recognised by cargo, and any additional files needed by the package. This is where cargo packages crates and this is where cargo publishes them.

Because a Cargo.toml file has extra metadata, like a package name, description, maintainability state, etc, I'd recommend we introduce a new target type, maybe, crate_export. I'd lean towards wrapping one or more rust_library targets, rather than accepting srcs and re-implementing (I think via macro) as java_export does - we control the providers of rust_library, and we can modify them as we need. We want a rust_library to be able to depend directly on a crate_export, so it should expose the same providers as rust_library does. We may need a separate proc_macro_crate_export target, or we may be able to get away with just an attribute on crate_export.

I'd probably start just wrapping a single rust_library, but being able to merge multiple may be important/useful in the future.

We'll need to generate a Cargo.toml file which is compatible with cargo. Our goal is roughly:

• Generate a Cargo.toml file
• Create a .tar.gz of that Cargo.toml file and any needed source files (and maybe resources too).
• Publish that .tar.gz file to crates.io (or some other registry).

Interesting challenges:

1. We need a source layout compatible with cargo. We can either generate a Cargo.toml a [lib] section that points at the "entry-point" source file (probably an attribute we'll need to select it by name, if a rust_library contains more than one file), or we can try to re-lay-out the source code. I'd recommend the former. This may get more complicated when we allow wrapping more than one rust_library in the same crate. I'd recommend not worrying about that for now.
2. Identifying dependencies. We should split dependencies into three kinds: "in-repo dependencies which are also published" (i.e. crate_export targets), "in-repo dependencies which are not published" (e.g. rust_library targets, but also things like cc_library targets), "external dependencies" (i.e. things pulled in by crate_universe). This means we'll need to introduce a way of detecting whether an in-repo dependency is/has a crate_export or just a rust_library (we can build this mechanism into the provider). I'd suggest for the initial cut, we just ban in-repo non-published dependencies. For in-repo published dependencies, and external dependencies, we need to identify the crates.io (or other registry) package name. For external dependencies, we currently generate tags with this information in which we can structure more if we need. For in-repo deps we'll need to be able to sniff this from a provider.
3. Picking version requirements and features to depend on. This is a little tricky. Ideally for external deps we probably want to use the same version requirement as was used in the crate_universe call that pulled it in? For internal deps, we probably want ^$version where $version already exists on rust_library. We may need to make this overridable on crate_export.
4. Non-rust dependencies, e.g. if someone depends on a cc_library, or a cargo_build_script. I'd suggest we just skip these for now.
5. Generated source files. For a first cut I'd probably just skip these, but we basically just need to generate them and copy them to their short_path (i.e. strip off their bazel-bin/... prefix).

Open questions:

1. Do we want to lay out the files in a directory and shell out to cargo to do the publish (we have access to a cargo binary on the rust_toolchain), or do we want to re-implement a publish binary ourselves which does the same logic? AIUI the exchange with crates.io is pretty simple, but it's pretty tempting to just shell out to cargo. We probably don't want to link cargo as a library, as it's not designed to be used as a library.
2. How should we accept auth tokens? (And AIUI cargo is currently adding new ways of doing auth, how many do we want to support?)

Concrete work items:

1. Write a rust binary which generates a Cargo.toml file for a rust_library target. This can be testesd independently from any other pieces by having a cargo project and trying to depend on a path dependency pointing at that Cargo.toml. This will require adding extra information to existing providers. I'd recommend starting by generating a completely hard-coded Cargo.toml file that you know is correct, then going back and replacing each of the hard-coded fields with information gained by traversing providers until there's nothing hard-coded any more. (e.g. hard-code a list of dependencies and their versions, then auto-detect the dependencies with hard-coded versions, then auto-detect the versions).
2. Generate a .tar.gz containing the generated Cargo.toml file and required source files (probably leaning on pkg_tar :)).
3. Publish the .tar.gz (either via our own rust binary, or by running cargo).

[Codssoar]

#vemtransformarcomagente

[flyingsilverfin]

Just came across this! I wanted to put out that we've independently come up with a solution to the problem of Bazel/Crate deployment interopability.

Granted, it works for a specific setup, but we've been using it successfully.

You can check the rules here: https://github.com/vaticle/bazel-distribution/blob/master/crates/rules.bzl

In short, we have a rule that converts a rust_library into a .crate archive, with a generated TOML manifest. It's not feature complete, but works for pretty sophisticated source packages so far, including handling crate universe and external bazel dependencies.

To solve dependencies on other bazel rust_library dependencies, we have to convert these into crate dependencies.

1. We expect these dependencies to have been published to the repository ahead of time.
2. All rust_library targets that we want to be cargo-compatible need a tags = ["crate-name=<crate name>"] tag.
3. We must pass what we call workspace_refs file into the crate assembly. In essence, we always depend on external Bazel workspaces via git "commit" or git "tag". "tag" is expected to be a release version. workspace_refs maps an external bazel workspace cloned via Git into a commit or tag version. During assembly, we look up the rust_library's external workspace, from which we look up the git tag version. We then get the crate-name tag from the library. Combined, these allow us to generate the correct Cargo dependency.

We currently upload to cargo/custom repository via an HTTP put, but will also consider farming this out to Cargo, since a hermitic Cargo executable is available in the rules_rust toolchains. However, we're currently just rolling with our current pair of assemble_crate and deploy_crate rules.

Simple example
Here's an example of a simple crate assembly/deployment with only crate universe dependencies.
https://github.com/vaticle/typeql/blob/6d42063bd15f9a293b0f3e19f082bd27a0df74a3/rust/BUILD#L58

Here's how we invoke it:
https://github.com/vaticle/typeql/blob/6d42063bd15f9a293b0f3e19f082bd27a0df74a3/.factory/automation.yml#L138

Note that we expect an environment variable to be used to pass the deployment token to the deployment script.

Complex example
Here's an example that uses both crate universe dependencies and bazel rust_library targets from an external workspace:
https://github.com/vaticle/typedb-driver/blob/d297991f9d18ae7365689f1b6f09057d9c6a4eb4/rust/BUILD#L72

with invocation:
https://github.com/vaticle/typedb-driver/blob/d297991f9d18ae7365689f1b6f09057d9c6a4eb4/.factory/automation.yml#L750

Note that this one uses the workspace_ref we generate in the WORKSPACE:
https://github.com/vaticle/typedb-driver/blob/d297991f9d18ae7365689f1b6f09057d9c6a4eb4/WORKSPACE#L246

Hopefully this is useful!