Adding cargo::no-new-rust-code to speed up compilation #915
Description
Proposal
motivation
Currently, all crates that bindgens, compiles or finds external dependencies have to wait for compilation/lookup of compiler artifacts to be done before compilation can continue.
For example, zstd-sys takes really a long time to compile zstd and all crates that depend on it have to wait until they are compiled.
Given that bindgen usually doesn't take that long (if user enabled zstd-sys/bindgen), the dependents of zstd-sys could start compilation earlier.
proposed solution
I propose that we introduce a new flag cargo::no-new-rust-code-change, it would tell cargo that the build script won't do anything that affects compilation of the rust code of this crate (generating code, issuing cfgs, etc), and so the rust crate can generate the rmeta and compile the code.
Once the rmeta is generated, dependents of this crate can start compiling without waiting for the whole compilation to be done.
If the build-script violates the protocol and actually tries to issue build-script instructions for cfgs then compilation would fail.
Generating code or changing already generated code that the rust crate includes!() would be considered the same as trying to modify the source code of the crate when rustc is compiling, rustc/cargo does not guarantee whether it would be read, and rustc could read the files when build script is updating it.
implementation
Implementing this would need -Zno-link/-Zlink-only to not rely on external static libraries produced by the build script, so some changes might be needed for it.
Q&A
how would that benefit systems with less than 4 cores, e.g. GHA?
It definitely cannot benefit single-core system, however for system with 2-4 cores it could still benefit.
build-script using cc-rs
P.S. I am one of the maintainers of cc-rs
For crate using cc-rs to compile the code, cc-rs cannot fully utilise every core at all time.
Particularly, cc-rs runs is_flag_supported logic in sequential, and that logic actually tries to compile a dummy c file to check if the compiler supports the flag.
cc-rs also has other places where it has to run external command sequentially to retrieve information (checking compiler family, looking for macOS/windows env, etc).
During parallel compilation (if feature cc/parallel is enabled), cc is not guaranteed to spawn a new job immediately when a job token is freed by another process (e.g. other build scripts, rustc, etc).
compilation using external systems
Compilation using external systems such as configure + make, or cmake (potentially through cmake-rs) typically involves a setup/env detection and makefile/ninja generation process that AFAIK is single thread and takes quite some time.
It might be enough for rustc to compile the crate itself and enable the dependents to be compiled in parallel.
And since ninja/make has an internal dependencies resolution process, it is not likely to fully utilise all cores at all time, unless the project doesn
Process
The main points of the Major Change Process are as follows:
- File an issue describing the proposal.
- A compiler team member who is knowledgeable in the area can second by writing
@rustbot secondor kickoff a team FCP with@rfcbot fcp $RESOLUTION.- Refer to Proposals, Approvals and Stabilization docs for when a second is sufficient, or when a full team FCP is required.
- Once an MCP is seconded, the Final Comment Period begins.
- Final Comment Period lasts for 10 days after all outstanding concerns are solved.
- Outstanding concerns will block the Final Comment Period from finishing. Once all concerns are resolved, the 10 day countdown is restarted.
- If no concerns are raised after 10 days since the resolution of the last outstanding concern, the MCP is considered approved.
You can read more about Major Change Proposals on forge.