Rust should integrate easily into large build systems

[aturon]

Overview

When working with larger organizations interested in using Rust, one of the first hurdles we tend to run into is fitting into an existing build system. We've been exploring a number of different approaches, each of which ends up using Cargo (and sometimes rustc) in different ways, with different stories about how to incorporate crates from the broader crates.io ecosystem. Part of the issue seems to be a perceived overlap between functionality in Cargo (and its notion of compilation unit) and in ambient build systems, but we have yet to truly get to the bottom of the issues—and it may be that the problem is one of communication, rather than of some technical gap.

By the end of 2017, this kind of integration should be easy: as a community, we should have a strong understanding of best practices, and potentially build tooling in support of those practices. And of course, we want to approach this goal with Rust's values in mind, ensuring that first-class access to the crates.io ecosystem is a cornerstone of our eventual story.

Projects

At this point, we are still trying to assess the problems people face in this area. If you have experience here, please leave a comment with your thoughts!

[luser]

I started a repo not long ago to collect anecdotes about people integrating Rust into existing projects. I only have a few examples in there (and I keep seeing more crop up all the time). It'd be great to at least do a survey of all the examples we can find of people attacking this problem to see what the major issues were.

From the Firefox perspective, we have a pretty custom build frontend that generates Makefiles in the backend, so we would have had to write the custom integration bits regardless. We did find that life got a lot better once we started invoking cargo instead of rustc directly. I'm sure there are projects that would get value out of having individual Rust source files in their codebase, but it feels like a lot of the value in the Rust ecosystem comes from using cargo and leveraging crates.io (I doubt this is contentious).

[goertzenator]

I've worked on build systems to compile Rust components for Erlang applications, so I'm in a good position talk about a few issues. An overview of the building that takes place can be found in this README. Erlang can make use of bins, dylibs, and cdylibs.

1. dylibs on OSX have special link requirements ("-- --codegen link-args='-flat_namespace -undefined suppress'") which creates a cascade of fussy work. Firstly, to provide that flag I need to "cargo rustc" instead of "cargo build", and to do that I need to detect all the binary/lib targets and build them one at a time. I really wish I could just "cargo build" and have cargo sort out the details for me. Maybe a "--extension-lib" flag for "cargo build" to apply this behavior? I understand this linking scenario is not unique to Erlang.

2. Discovering and locating the output files is tricky. I have to "cargo read-manifest" to find the targets, form a platform-specific name from those results, then parse the flags for any "--target" flags to form a path for these files. I would love a flag of the form "--print-artifacts=[bin|lib|dylib|cdylib]" for "cargo build" to print the full output path and name to stdout.

[bobsomers]

This is great. Integrating Rust/Cargo with Bazel would be the first major hurdle to us using more Rust in our codebase at work, which is a pretty large mass of C++ code with some Python sprinkled throughout.

Since both Bazel and Cargo are fairly opinionated about how builds and package management should work, and since I am an expert in neither, it's not immediately clear to me which build system should be doing what or if we should just try to integrate rustc into Bazel without Cargo at all. Using strictly Cargo is (unfortunately) probably out of the question since most of the C++ and Python packages are dependency tracked with Bazel BUILD files, and any serious integration with the C++ code would require our Rust libraries and binaries to be dependency tracked by Bazel as well.

[steveklabnik]

I thought Bazel already had basic support for rustc? Maybe my info is outdated though.

…

On Mon, Feb 6, 2017 at 5:51 PM, Bob Somers ***@***.***> wrote: This is great. Integrating Rust/Cargo with Bazel <https://bazel.build/> would be the first major hurdle to us using more Rust in our codebase at work, which is a pretty large mass of C++ code with some Python sprinkled throughout. Since both Bazel and Cargo are fairly opinionated about how builds and package management should work, and since I am an expert in neither, it's not immediately clear to me which build system should be doing what or if we should just try to integrate rustc into Bazel without Cargo at all. Using strictly Cargo is (unfortunately) probably out of the question since most of the C++ and Python packages are dependency tracked with Bazel BUILD files, and any serious integration with the C++ code would require our Rust libraries and binaries to be dependency tracked by Bazel as well. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABsipZo1JTHfFy5idL4sB97BOsZPUT1ks5rZ6P3gaJpZM4Ly9Nx> .

[mfarrugi]

Bazel has support for local sources, which is to say it does not support crates.io. There's an issue open for it, but the project is relatively inactive bazelbuild/rules_rust#2. One would need to keep a local repository clone of a crate and all its dependencies to make bazel happy without modifications.

Bazel's rust targets also can't be depended on by c/c++ at the moment.

The kythe project has wrappers for bazel to call cargo, but I it's not the most robust approach to integration. For reference, kythe:tools/build_rules/rust.

disclaimer: I don't know too much about bazel, happened to learn the above looking into it last week.

[withoutboats]

Bazel's been mentioned several times this has come up. I believe Dropbox needs to integrate its Rust into a Bazel build process, and I think I might have been told Facebook uses it as well (but it might also have been that Facebook has an internal tool is similar to Bazel). It seems like a promising tool to look into for this issue.

[davidzchen]

Bazel Rust Rules author here. Apologies for the inactivity on the rules_rust project; I have been busy with other projects on my plate. I am planning to implement the workspace rules for pulling crates from Cargo (bazelbuild/rules_rust#2) by the end of Q1 with a stretch goal of also implementing tooling for automatically generating Bazel BUILD files from Cargo.toml (bazelbuild/rules_rust#3), though the latter will likely extend into Q2.

The Kythe project has rules that shell out to Cargo directly as quick stop-gap measure, but those rules are meant to be used internally in Kythe for now (since they are not hermetic for instance) and the plan is to replace them with the rules in rules_rust once features such as pulling from Cargo are supported.

Of course, if anyone is interested in helping with improving the Bazel rules for Rust, contributions are certainly welcome. :)

+cc @damienmg

[shahms]

Kythe contributor here. We'd definitely like to see better integration with Cargo from the upstream Bazel Rust rules. Our extant integration was very much a hack to allow our intern to make progress on the Rust indexer itself, rather than getting bogged down with Bazel integration.

[LegNeato]

Facebook uses Buck, and there is some early Rust support:

https://buckbuild.com/rule/rust_binary.html

Facebook vendors their dependencies in-tree.

[jsgf]

Facebook uses Buck, and there is some early rust support:

I've spent quite a bit of time on that over the last few months, and it's getting pretty solid now. It's well integrated with the overall build/test system and (most recently) can also interop with cxx rules.

[jpakkane]

I'm the main author of Meson build system that is being used by GStreamer and a bunch of other projects. We also have Rust support, which is a bit rudimentary but can be used to build stuff like a Python extension module that uses C, C++, Rust and Fortran in a single target. We aim to improve Rust support. This is especially important for mixed-language projects, since Cargo is nice for plain Rust projects but I'm fairly certain that Cargo developers do not want to add first class multiplatform C/C++ build support to Cargo.

[larsbergstrom]

A key part of doing this well will involve handling updates to the dependency graph in a deliberate and piecemeal way, particularly in scenarios where upstream "master" has moved to a new version (e.g., as in rust-lang/cargo#2649).

We (@nox, @SimonSapin, and I) have had many conversations with @alexcrichton and @wycats on this front, and I believe the leading contender from their point of view is for some extensions to fix up paths and avoid the abuse of [replace], as it runs immediately into version-related roadblocks for any non-trivial project.

[moretea]

Nix (which has a similar pure view of build systems as Bazel) uses a trick that involves cloning a well known version of the crates index git repository, see https://github.com/NixOS/nixpkgs/tree/master/pkgs/build-support/rust

[sjmackenzie]

Sadly the nixpkgs crates index is not hermetically sealed, it gave us many problems. So we implemented a crate index nixifier, which reads crates.io-index and spits out a nixfied crates index. This allows one to use nix to completely manage transitive crate dependencies of a project without needing cargo. The repo: https://github.com/fractalide/nixcrates