Incorrect branch targets in RISC-V executables built with LTO

[ilovepi]

We're seeing several cases where branch targets are invalid, for example in the middle of another instruction, when building Fuchsia's zircon kernel with flto=full. This only seems to occur under -flto-full, and both non-LTO and ThinLTO do not seem to have this issue.

Most branches appear to be correct, or at least have valid targets, but in some cases the target isn't the location of a valid instruction. This sounds like it could be related to linker relaxation, but https://discourse.llvm.org/t/riscv-status-of-lto-for-risc-v/58518 isn't clear on the current status of LTO for RISC-V, so it may be related.

I'm working on getting something smaller than our kernel image as a reproducer, but in the meantime I'm providing a snippet from a relevant case. The target for the bne instruction is in the middle of another instruction sequence

  1b2ac8:	1d051363          	bne	a0,a6,1b2c8e <VmCowPages::AddNewPageLocked(unsigned long, vm_page*, VmCowPages::CanOverwriteContent, VmPageOrMarker*, bool, bool)+0x1ea>
...
  1b2c80:	0d000593          	li	a1,208
  1b2c84:	000ac097          	auipc	ra,0xac
  1b2c88:	8a8080e7          	jalr	-1880(ra) # 25e52c <assert_fail>
fbl::Canary<1447904080u>::Assert() const:
./../../zircon/system/ulib/fbl/include/fbl/canary.h:64
  1b2c8c:	030aa783          	lw	a5,48(s5)
  1b2c90:	ffea5517          	auipc	a0,0xffea5

https://fxbug.dev/129493 has additional context.

CC: @topperc @MaskRay @asb

[llvmbot]

@llvm/issue-subscribers-backend-risc-v

[kito-cheng]

The guideline for those kind of bug in my mind is: could you try to disable linker relaxation at all to see if it work? I am not sure how to disable that correctly at LTO build, but you could try to add -mno-relax at CFLAGS and LDFLAGS=-Wl,--no-relax?

And if it not work...@MaskRay could help :P

[jrtc27]

Probably something like -Wl,--plugin-opt=-mattr=-relax

[MaskRay]

#59350 (comment) this may be relevant

If you use clang++ --target=riscv64-linux-gnu -flto -fuse-ld=lld a.cc -r -o a.o -Wl,-mllvm,-mattr=+c,-mllvm,-mattr=+relax, you'll get R_RISCV_RELAX. If you omit the -Wl, then there will be no R_RISCV_RELAX, and e_flags of the ELF header does not contain EF_RISCV_RVC. This is due to a target-features LTO issue which hasn't been resolved.

[nickdesaulniers]

@twd2 spotted a case that smells a lot like fc018eb.

If we need to re-pass compiler flags to the linker, but only for LTO, that is a deficit in the IR IMO.

[petrhosek]

We have now also encountered this issue in ThinLTO builds. We no longer think it's LTO related, our current working theory is that it's a garbage-in-garbage-out scenario where the backend generates branches with invalid targets and LLD fails to detect and warn for this case. We believe it might be related to the size of module which is why it's manifesting primarily in LTO and ThinLTO builds, but we're trying to see if we could reproduce it in a regular build as well. While we're trying to narrow down the issue, I think it'd be helpful if LLD could diagnose this case and give a warning.

[twd2]

We have now also encountered this issue in ThinLTO builds. We no longer think it's LTO related, our current working theory is that it's a garbage-in-garbage-out scenario where the backend generates branches with invalid targets and LLD fails to detect and warn for this case. We believe it might be related to the size of module which is why it's manifesting primarily in LTO and ThinLTO builds, but we're trying to see if we could reproduce it in a regular build as well. While we're trying to narrow down the issue, I think it'd be helpful if LLD could diagnose this case and give a warning.

Please check ClangBuiltLinux/linux#1942 (comment) to see if this is similar to your case :)