Implement "pipelined" rustc compilation (compiler side)

[nikomatsakis]

@alexcrichton opened rust-lang/cargo#6660, which describes a scheme to do "pipelined" compilation. This issue represents the rustc side of that work. I'm opening it -- initially, at least -- so that I can nominate the issue and we can discuss in the compiler team meetings. It may then serve as the hub for any mentoring re: the rustc part of the equation.

I would like us to decide whether to do this work and how to integrate that into our overall planning (who will do the rustc part of the work? How hard is it? etc)

[alexcrichton]

For the rustc side of things I believe there's two primary work items to implement:

• The compiler should be able to produce an rlib when the dependencies are only available as rmeta files. Currently when producing an rlib I believe the compiler requires all dependencies to be an rlib, but all dependencies only need to be an rlib if we're producing a final linkable artifact. When an rlib is produced only the metadata needs to be fundamentally available for all dependencies.

• Somehow the compiler needs to signal to Cargo when the rmeta file for a compilation is ready to go. The rmeta file would be produced as soon as possible in compilation and then Cargo would be signaled to its existence so Cargo could continue compilation of later dependencies if necessary. The exact mechanism to do this is debated in Implement "pipelined" rustc compilation cargo#6660, but the current to main candidates are: send a message to a TCP server hosted by Cargo, or print something to stdout which Cargo will read

[ehuss]

Currently when producing an rlib I believe the compiler requires all dependencies to be an rlib

I'm able to get around this with -Z always-encode-mir (it seems to successfully create the rlib from rmeta deps), but when linking the final output with these rlibs, I get crate foo required to be available in rlib format, but was not found in this form. Is there a way to tell rustc how to pick up the rlibs for transitive deps? Or is that embedded in the rlib with no way to override?

[alexcrichton]

I think that -Z always-encode-mir being required is a bug and I suspect that the other error message is likely indicative of the same bug. While it may be possible to work around, I think we'll definitely want to change rustc and how it loads dependencies to fully fix this. I think today it's basically not possible to emulate the behavior we want

[agausmann]

My first instinct would be to make rmeta generation a subcommand of the compiler, with the signal being the termination of the process. After that, the compiler can be re-invoked to create the rlib. The feasibility of that would depend on how much compiler state gets stored to disk during rmeta creation, and how much would get lost on exit that would have to be recreated on the second run, which I don't know much about.

[nnethercote]

The produce-an-rlib-from-rmeta-inputs part needs to be done first and sounds like the larger piece of work. I would like to work on that.

rust-lang/cargo#6660 has some discussion of the cargo/rustc interaction, pros and cons of different approaches.

[alexcrichton]

So it turns out I was remember a time long since passed, and my first bullet above is seemingly already done (@nnethercote is going to fully confirm). That means that actually all rustc needs to do is to somehow signal Cargo when a file is ready to go!

I've talked with @nnethercote about this and I think he's going to work on implementing the JSON message idea (where rustc emits a JSON message whenever a file is ready for Cargo to consume).

@ehuss FWIW rustc generates different rmeta files if you do --emit metadata vs if you do --emit metadata,link, so I think that's why you needed -Z always-encode-mir. Cargo would, however, always execute --emit metadata,link,dep-info, so I think we're covered!

[ehuss]

Oh, indeed, it does seem to work!

[pnkfelix]

There is now WG-pipelining: https://github.com/rust-lang/compiler-team/tree/master/working-groups/pipelining

I'm going to remove the nominated tag from this ticket.

I don't know whether this ticket should be repurposed into a tracking issue for this work, or simply be closed. I'll leave that choice up to WG-pipelining.

[nnethercote]

#60006 just landed, which implements the JSON message rustc gives to Cargo so that Cargo knows when metadata has been written.

I'm now working on #60385, which will move metadata writing much earlier in the pipeline -- currently it happens right at the end, preventing much in the way of crate overlap that is the whole goal of pipelining.

[alexcrichton]

I've opened a dedicated tracking issue for stabilizing -Z emit-directives