1.43.0 x.py "dist" build fails in LlvmTools build

[he32]

Since I'm building bootstrap kits for rust, I need to use the "dist" argument to
the x.py script, and in 1.43.0 this now fails in the llvm build with

Dist RLS stage1 (x86_64-unknown-netbsd)
running: "/usr/pkgsrc/lang/rust/work/rustc-1.43.0-src/build/x86_64-unknown-netbsd/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=RLS-ready-to-serve." "--image-dir" "/usr/pkgsrc/lang/rust/work/rustc-1.43.0-src/build/tmp/dist/rls-image" "--work-dir" "/usr/pkgsrc/lang/rust/work/rustc-1.43.0-src/build/tmp/dist" "--output-dir" "/usr/pkgsrc/lang/rust/work/rustc-1.43.0-src/build/dist" "--non-installed-overlay" "/usr/pkgsrc/lang/rust/work/rustc-1.43.0-src/build/tmp/dist/rls-overlay" "--package-name=rls-1.41.0-x86_64-unknown-netbsd" "--legacy-manifest-dirs=rustlib,cargo" "--component-name=rls-preview"
        finished in 16.024
  < Rls { compiler: Compiler { stage: 1, host: "x86_64-unknown-netbsd" }, target: "x86_64-unknown-netbsd" }
  > LlvmTools { target: "x86_64-unknown-netbsd" }
Dist LlvmTools (x86_64-unknown-netbsd)
thread 'main' panicked at 'Error: File "/usr/pkgsrc/lang/rust/work/rustc-1.43.0-src/build/x86_64-unknown-netbsd/llvm/bin/llvm-nm" not found!', src/bootstrap/lib.rs:1234:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
        finished in 0.006
Traceback (most recent call last):
  File "./x.py", line 11, in <module>
    bootstrap.main()

The llvm build has left llvm-nm in
work/rustc-1.43.0-src/build/x86_64-unknown-netbsd/llvm/build/bin/llvm-nm
but that's apparently not where the build expects to find the result.

Building 1.43.0 with the "build" target to x.py succeeds, but of course does not
accomplish building the bootstrap kits.

[Mark-Simulacrum]

Since this did work on our dist builders, presumably, I'm a bit surprised to see this. Are you applying any patches atop our build? Maybe LLVM has some internal detection which isn't running? (Can you see if the llvm/bin dir, not in build is being created with x.py build? It seems like we expect that to be the case.)

[he32]

Since this did work on our dist builders, presumably, I'm a bit surprised to see this. Are you applying any patches atop our build?

Yes, I'll have to admit that we do. https://github.com/NetBSD/pkgsrc/tree/trunk/lang/rust is our set of build scaffolding and patches to make this build on NetBSD. This worked with 1.42.0.

Maybe LLVM has some internal detection which isn't running? (Can you see if the `llvm/bin` dir, not in build is being created with `x.py build`? It seems like we expect that to be the case.)

Hmm... I find an llvm/bin/llvm-nm in d3: {4} find work -path '*/llvm/bin/llvm-nm' work/.destdir/usr/pkgsrc/lang/rust/work/rustc-1.43.0-src/build/x86_64-unknown-netbsd/llvm/bin/llvm-nm d3: {5} So parts of the build appears to respect ${DESTDIR}, but some other part doesn't(?) Regards, - Håvard

[Mark-Simulacrum]

Hm, so I'm not seeing anything initially suspicious at a quick glance through. We ourselves don't set DESTDIR, though, and maybe that's being threaded through to the LLVM build and causes trouble?

AFAICT, x.py dist just uses the existing pre-built llvm -- I would expect that if x.py build worked (i.e. created the file where x.py dist expected it) x.py dist would just work too.

Did you start setting DESTDIR since 1.42 (or perhaps something else in your environment changed)?

I pretty much see nothing suspicious on our end in the diff between 1.42 and 1.43 (at least in src/bootstrap). LLVM didn't even have any significant changes between those two tags as best as I can tell...

[he32]

Hm, so I'm not seeing anything initially suspicious at a quick glance through. We ourselves don't set DESTDIR, though, and maybe that's being threaded through to the LLVM build and causes trouble? AFAICT, x.py dist just uses the existing pre-built llvm -- I would expect that if x.py build worked (i.e. created the file where x.py dist expected it) x.py dist would just work too.

I'm using "dist" instead of "build", not doing "dist" after first doing "build".

Did you start setting DESTDIR since 1.42 (or perhaps something else in your environment changed)?

DESTDIR is always set when building in pkgsrc, so it's always been set earlier.

I pretty much see nothing suspicious on our end in the diff between 1.42 and 1.43 (at least in src/bootstrap). LLVM didn't even have any significant changes between those two tags as best as I can tell...

Hmm... - Håvard

[Mark-Simulacrum]

I'm using "dist" instead of "build", not doing "dist" after first doing "build".

Yes, dist internally just "runs" build (not on the command line, but the same effect, I think).

Yeah, I guess -- one thing that could be helpful is to try to bisect this? I imagine it would take a long time though, given how slow building the compiler can be.

[he32]

It looks like this problem has been there for a while; it's also present in 1.41.1,
and probably in earlier versions as well (I didn't check earlier versions).

I've replicated it outside of the pkgsrc setup, and it's indeed being triggered by
a non-null DESTDIR in the environment passed to x.py when doing the "dist" target.
I have a workaround for the pkgsrc context by unset'ing DESTDIR.

However, that workaround apparently has the unfortunate consequence that
the time to install suddenly got significantly prolonged, because cargo detects
a difference in the environment between the "build" and "install" phases, and
decides that "everything needs a re-build on install".

[Mark-Simulacrum]

Ah, okay. Sounds like we probably need to unset DESTDIR internally when building LLVM? I'd be happy to accept a PR for that, but probably won't get to it myself.

[jyn514]

I'm going to close this in favor of #73132.