Latest master toolchains are broken when linking LLVM: "__morestack: invalid needed version 2"

[RalfJung]

When I install the latest master toolchains via rustup-toolchain-install-master and try to build Miri with them, linking fails:

error: linking with `cc` failed: exit code: 1
  |
  [...]
  = note: /usr/bin/ld: /home/r/.rustup/toolchains/miri/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-11-rust-1.51.0-nightly.so: __morestack: invalid needed version 2
          /usr/bin/ld: /home/r/.rustup/toolchains/miri/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-11-rust-1.51.0-nightly.so: error adding symbols: bad value
          collect2: error: ld returned 1 exit status

I did some bisection, and it looks like the problem started happening with #81489.

Cc @rust-lang/infra

[RalfJung]

The error is actually already mentioned in #81489. However, I am building Miri with cargo, so I don't know how I'd "use LLD instead".

[Mark-Simulacrum]

Hm, we probably need to revert that PR. I did not realize we'd need lld downstream too, I thought that was only on our builder.

cc @nikic @cuviper

[cuviper]

Yes, a revert seems appropriate. We're going to have to figure out where that's actually coming from. The only change I saw between 10 and 11 by grepping morestack was the removal of llgo, but maybe there's some bad remnant.

[nikic]

Yeah, I thought this issue would be limited to the rustc build, but it makes sense that it can also affect project linking against rustc shared objects.

I've looked a bit into what the root cause is, and what I have so far as that the symbol comes from https://github.com/llvm/llvm-project/blob/b10d445307a0f3c7e5522836b4331090aacaf349/llvm/lib/ExecutionEngine/RuntimeDyld/RTDyldMemoryManager.cpp#L217 and is present as a weak symbol both when compiling with clang-10 and clang-11. The only difference I see from objdump -T is this:

; clang-10
0000000000000000  w   D  *UND*  0000000000000000  LLVM_11     __morestack
; clang-11
0000000000000000  w   D  *UND*  0000000000000000  Base        __morestack

So apparently __morestack was treated as a symbol of the LLVM_11 library, while now it's treated as a symbol of Base -- I'm not sure what that actually means, but presumably that refers to the compiler runtime libraries. My guess is that previously it just ended up being nulled out as a missing weak symbol, but now it's actually being found and linking fails because ... reasons. I'm not good at linkers :)

Edit: readelf -Ws shows these as:

; clang-10
   315: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __morestack
; clang-11
   315: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __morestack@@<corrupt>

[RalfJung]

@nikic can you prepare a revert PR?

[nikic]

@RalfJung Done in #81565.

If nobody has an idea on how to avoid the linker error, two more options would be to comment out the symbol in our llvm-project fork, or to avoid upgrading clang altogether and just build both Python 2 and Python 3 for that image.

[nagisa]

The issue is probably going to be in how exactly the objects that comprise libLLVM.so are compiled, not linked. I suspect lld just ends up ignoring the malformed object or otherwise not reporting the issue, but the object and the relevant symbols ultimately end up in the so anyway.

Alas, reasons these obscure failures can occur are many – I've seen them occur from things as trivial as forgetting to -fPIC.

The question is: is there perhaps a way to disable ExecutionEngine entirely? Intuitively nothing we do should depend on that code. And if not, I'm comfortable patching out __morestack in our fork as we get closer to Feb 11 and continue having trouble pinpointing exact cause of this problem.

[nikic]

The question is: is there perhaps a way to disable ExecutionEngine entirely? Intuitively nothing we do should depend on that code.

We normally don't link ExecutionEngine, but dist-x86_64-linux in particular builds LLVM with ThinLTO, in which case we link libLLVM.so as shared, instead of statically linking individual components into rustc_llvm.