"Can't find crate for `rustc_*`" when cross-compiling a crate with `rustc_private` on

[momvart]

Problem

In our project, we are using rustc components. When building for a x86_64 target it's fine. However, when setting the target to i686-unknown-linux-gnu, cargo (rustc) starts complaining about components not existing.

I think this is a problem due to components installation through rustup. It doesn't look to be installing them for the specified target.

The initial install is done through the following command.

$> rustup toolchain install nightly-2023-01-26 --target i686-unknown-linux-gnu --component rust-src rustc-dev llvm-tools-preview 
info: syncing channel updates for 'nightly-2023-01-26-x86_64-unknown-linux-gnu'
info: latest update on 2023-01-26, rust version 1.69.0-nightly (c18a5e8a5 2023-01-25)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'llvm-tools'
info: downloading component 'rust-docs'
info: downloading component 'rust-src'
info: downloading component 'rust-std' for 'i686-unknown-linux-gnu'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 68.3 MiB /  68.3 MiB (100 %)  45.8 MiB/s in  1s ETA:  0s
info: downloading component 'rustc-dev'
111.1 MiB / 111.1 MiB (100 %)  44.5 MiB/s in  2s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'llvm-tools'
 32.8 MiB /  32.8 MiB (100 %)  23.6 MiB/s in  1s ETA:  0s
info: installing component 'rust-docs'
 19.4 MiB /  19.4 MiB (100 %)  17.7 MiB/s in  1s ETA:  0s
info: installing component 'rust-src'
info: installing component 'rust-std' for 'i686-unknown-linux-gnu'
 28.1 MiB /  28.1 MiB (100 %)  20.7 MiB/s in  1s ETA:  0s
info: installing component 'rust-std'
 30.1 MiB /  30.1 MiB (100 %)  20.5 MiB/s in  1s ETA:  0s
info: installing component 'rustc'
 68.3 MiB /  68.3 MiB (100 %)  23.2 MiB/s in  2s ETA:  0s
info: installing component 'rustc-dev'
111.1 MiB / 111.1 MiB (100 %)  21.6 MiB/s in  5s ETA:  0s
info: installing component 'rustfmt'

  nightly-2023-01-26-x86_64-unknown-linux-gnu installed - rustc 1.69.0-nightly (c18a5e8a5 2023-01-25)

However, during building rustup complaints as follows.

$> cargo build --target=i686-unknown-linux-gnu
...
error[E0463]: can't find crate for `rustc_middle`
  --> compiler/src/lib.rs:19:1
   |
19 | extern crate rustc_middle;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate
   |
   = help: maybe you need to install the missing components with: `rustup component add rust-src rustc-dev llvm-tools-preview`

Trying to install components again raises a conflict issue.

$> rustup component add --target i686-unknown-linux-gnu rust-src rustc-dev llvm-tools-preview  
info: component 'rust-src' is up to date
info: downloading component 'rustc-dev' for 'i686-unknown-linux-gnu'
info: installing component 'rustc-dev' for 'i686-unknown-linux-gnu'
128.4 MiB / 128.4 MiB (100 %)  21.6 MiB/s in  5s ETA:  0s
info: rolling back changes
error: failed to install component: 'rustc-dev-i686-unknown-linux-gnu', detected conflict: 'lib/rustlib/rustc-src/rust/Cargo.lock'

Originally, we are using rust-toolchain.toml to define the components and targets. In both cases, these issues exist. Also tried installing it side by side the x86_64 target and the same issue exists.

Steps

1. Create a project using rustc components.
2. Create a rust-toolchain.toml to override the toolchain and define components and targets as follows.

[toolchain]
channel = "nightly-2023-01-26"
components = ["rust-src", "rustc-dev", "llvm-tools-preview"]
targets = ["i686-unknown-linux-gnu"]

3. Build using cargo build --target=i686-unknown-linux-gnu.

Possible Solution(s)

No response

Notes

No response

Rustup version

rustup 1.25.2 (17db695f1 2023-02-01)

Installed toolchains

Default host: x86_64-unknown-linux-gnu
rustup home:  ...

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu
nightly-2023-01-26-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)

installed targets for active toolchain
--------------------------------------

i686-unknown-linux-gnu
x86_64-unknown-linux-gnu

active toolchain
----------------

nightly-2023-01-26-x86_64-unknown-linux-gnu (overridden by '.../rust-toolchain.toml')
rustc 1.69.0-nightly (c18a5e8a5 2023-01-25)

[rbtcollins]

What does rustup component list --installed output? And rustup target list --installed ?

Then, the contents of each of ~/.rustup/toolchains/*/lib/rustlib/components

[momvart]

What does rustup component list --installed output?

$>  rustup component list --installed
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
llvm-tools-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-src
rust-std-i686-unknown-linux-gnu
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustc-dev-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu

And rustup target list --installed ?

$> rustup target list --installed
i686-unknown-linux-gnu
x86_64-unknown-linux-gnu

Then, the contents of each of ~/.rustup/toolchains/*/lib/rustlib/components

Sorry. I didn't get this one. Do you want the manifest files contents?

[rbtcollins]

Rustup writes to the file called components matching that glob.

[momvart]

Rustup writes to the file called components matching that glob.

Yeah but I didn't get the each part. Anyways, this is the components file content.

~/.rustup/toolchains/nightly-2023-01-26-x86_64-unknown-linux-gnu/lib/rustlib$ cat components 
cargo-x86_64-unknown-linux-gnu
clippy-preview-x86_64-unknown-linux-gnu
llvm-tools-preview-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-src
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustc-dev-x86_64-unknown-linux-gnu
rustfmt-preview-x86_64-unknown-linux-gnu
rust-std-i686-unknown-linux-gnu

[Stroopwafe1]

I also had this issue where trying to cross-compile to i686-unknown-linux-gnu failed to find the std and core crates. Even though I used the rustup target add ... command. Turns out that after switching to the stable channel, and adding it again it now works.

Somehow using nightly messes up cross-compilation

[tombh]

I'm seeing this issue too, but can't find a workaround. Neither manually adding components nor switching to stable, re-adding and switching to nightly, seem to help.

I actually have a public repo that can be used to reproduce this, here's a permalink to a failing Github Action workflow: https://github.com/tombh/rust-gpu-cli/blob/4a2492a95759cf103f4c2f3f9e51a1c7d0866b06/.github/workflows/release.yml#L45 And a failing workflow run's logs: https://github.com/tombh/rust-gpu-cli/actions/runs/11373072910/job/31638834293#step:5:373

[rami3l]

I'm seeing this issue too, but can't find a workaround. Neither manually adding components nor switching to stable, re-adding and switching to nightly, seem to help.

I actually have a public repo that can be used to reproduce this, here's a permalink to a failing Github Action workflow: https://github.com/tombh/rust-gpu-cli/blob/4a2492a95759cf103f4c2f3f9e51a1c7d0866b06/.github/workflows/release.yml#L45 And a failing workflow run's logs: https://github.com/tombh/rust-gpu-cli/actions/runs/11373072910/job/31638834293#step:5:373

@tombh Thanks for the ping! However, It looks like you're not describing a toolchain corruption (the symbolic "failed to install component" message is missing from your log), but another scenario which is related to RustGPU usage specifically.

I've found EmbarkStudios/kajiya#78 which looks similar, but I'm not completely sure what has caused the issue for you, especially since you're cross-compiling for another target... Is there any prior art in building RustGPU with cross?

PS: You have a typo in your script, it's --target aarch64-unknown-linux-gnu, not aarch63.

[tombh]

Thanks for looking into this!

So if I'm understanding the original bug report the issue arises before, and indeed can be reproduced without, the failed to install component error. Copying from the parent comment, this is the error that both @momvart and I are seeing:

$> cargo build --target=i686-unknown-linux-gnu
...
error[E0463]: can't find crate for `rustc_middle`
  --> compiler/src/lib.rs:19:1
   |
19 | extern crate rustc_middle;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate
   |
   = help: maybe you need to install the missing components with: `rustup component add rust-src rustc-dev llvm-tools-preview`

In other words, this is how I believe the error can be stated: there appears to be no way in which rust-src can be installed such that cross-compilation works on nightly Rust.

I agree that the subsequent error (ie failed to install component) is both an understandable next step (heeding the advice of the hint) and a reasonable response from rustup (either there is a further double-install bug, or there is a conflict with rust-toolchain.toml). But this is not the error in question.

https://github.com/Rust-GPU/rust-gpu is the original, underlying repo. The thing to note is that the core of the project is a custom codegen backend, which is what necessitates the rust-src, rustc-dev, llvm-tools components. And because it's still in beta, they've never tried to release any packages, and so have never ventured into cross-compiling.

Ah yes, that was a typo in aarch63, but it was just a debugging line. I removed it and still get the same error: https://github.com/tombh/rust-gpu-cli/actions/runs/11386877927/job/31680155871#step:5:373

[rami3l]

@tombh Thanks for your reply. Now that I think about it, it's less likely an issue specific to RustGPU, but common to nightly + rustc_private + cross compilation. I'm still not sure what has caused it: I mean, is it the component, the distribution, Rustup itself, or some sort of config / extern state?

Anyway, I'll find some time to do more investigation.

Possibly related: os-checker/MIRAI#3