Generating entire crates during build.rs

[nipunn1313]

Describe the problem you are trying to solve
When having entire autogenerated crates (eg https://github.com/dropbox/pb-jelly) - there isn't a good way to generate the crate at build time via build.rs.

Eg when building https://github.com/dropbox/pb-jelly/blob/a0c0550/examples/Cargo.toml, we get

nipunn-mbp:examples nipunn$ cargo build
error: failed to read `/Users/nipunn/src/pb-jelly/pb-test/gen/pb-jelly/proto_pbtest/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

This is because proto_pbtest is generated in the build phase.

Current workarounds include

• commenting out the proto_pbtest dependency, building, uncommenting, building again
• Creating a dummy empty proto_pbtest Cargo.toml to be overwritten
• Splitting the build phase out into a separate crate

Describe the solution you'd like

Idea 1

• Parse the Cargo.toml - tolerating missing path dependencies
• Build the build.rs - only looking at build dependencies - ensuring they're all available
• Run build.rs
• Re-evaluate cargo task graph to search for the missing dependencies
• Run rest of the build

Idea 2

• Provide a command (eg cargo build --prebuild-only) to only run the build.rs build phase tolerating missing path dependencies in the other phases
• Developer can run cargo build --prebuild-only followed by cargo build for projects like this
  One idea, may be to

Idea 3 [hacky workaround]

• Provide a mode to tolerate missing path dependencies cargo build --tolerate-missing-path-deps
• Developer can run cargo build ; cargo build

Notes

[nipunn1313]

Note that with idea 1 - if the build.rs updates some existing generated crates - those would need to be rebuilt.

Eg

[dependency]
generated_crate = {path = "/path/to/generated/crate"}

We'd need to make sure that the build.rs runs before generated_crate is compiled.

Perhaps there's some other way of achieving this. I see in the docs there is a section for code generation https://doc.rust-lang.org/cargo/reference/build-script-examples.html#code-generation - but it does not seem to include generating entire crates in this way.

[alexcrichton]

Most of Cargo unfortunately isn't really architected to support a use case like this. Injecting dependencies or updating the crate resolution graph dynamically during build time would require a lot of refactoring internally inside of Cargo. This is likely a situation where a non-Cargo wrapper will need to be used to prepare crates to get built by Cargo.

[nipunn1313]

Am curious if something like Idea-2 would be easier based on how cargo is architected?

Then the non-Cargo wrapper would be pretty simple (something like this)

cargo build --build-only
cargo build

It would be nice if the non-cargo-wrapper could execute a pair of cargo commands based out of a single Cargo.toml.
Currently, it requires a separate Cargo.toml for the build phase (its own crate) for the build-autogenerated-crates step.

Thanks for taking a look!

[alexcrichton]

Unfortunately I think even that would be pretty difficult. One issue is that every execution of Cargo needs to perform version resolution and figure out the crate graph. In a hypothetical --build-only mode it would need to understand that the crate graph is incomplete and somehow reflect that in Cargo.lock. Additionally build dependencies can be arbitrarily nested so the --build-only process would be iterative depending on your crate graph. Overall this I think would require intertwining compiling code and resolving the crate graph, which would be a very significant undertaking in Cargo for sure.

[notdanilo]

@nipunn1313 did you find a workaround for it? I am also trying to dynamically generate a crate and I want to be able to include it.

[notdanilo]

@alexcrichton is there any way to trick Cargo to:

1. Think that a dependency is cached (even though it isn't present in crates.io, it's located and findeable in the .cargo registry)?
2. Link the crate using a CLI arg without listing it in [dependencies]?

[nipunn1313]

workaround I found is to write a separate (not build.rs) script to dynamically generate the crate and run it before cargo build.

Separate script could even be written in rust, but the build has to be in two phases.
This does it for example - https://github.com/dropbox/pb-jelly/tree/main/pb-jelly-gen

[notdanilo]

@nipunn1313 right! I just noticed you have a different requirement.

pb-jelly-gen is included in [dependencies] already and it's patched in [patch.crates-io].

I am looking for a way to generate and include the project in a project without modifying its Cargo.toml.

My goal is to integrate pip to Cargo

[package.metadata.pip]
diffusers = "0.28.2"

The tool is supposed to download the package, parse it, generate the Rust code in a Cargo project and include it in the same project that uses the [package.metadata.pip].

[nipunn1313]

yeah - it will be tough to do in a single cargo build command.

You might be able to do something like this

Make a pip-gen crate in a subdirectory

cd pip-gen
cargo run

Then go to parent directory and build

cargo build