chore: Update to LLVM 19

[vadorovsky]

Latest Rust nightly switched to LLVM 19, so we need to update in order to support rustc-llvm-proxy.

---

This change is 

[tamird]

Reviewed 5 of 5 files at r1, 2 of 2 files at r2, 1 of 1 files at r3, all commit messages.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @vadorovsky)

---

-- commits line 12 at r2:
please include rationale

---

Cargo.lock line 84 at r1 (raw file):

 "libc",
 "libloading",
 "llvm-sys 180.0.0",

we now have two versions of llvm-sys, is that what we want?

---

.github/workflows/ci.yml line 121 at r1 (raw file):

          sudo apt update
          sudo apt -y install llvm-${{ matrix.llvm }}-dev libpolly-${{ matrix.llvm }}-dev

why?

---

.github/workflows/ci.yml line 153 at r1 (raw file):

      - name: Check
        if: matrix.llvm == 'source'
        run: cargo hack check --feature-powerset --features llvm-sys/force-dynamic

just wondering: do we need to force it at all? can we just let it pick? the LLVM we build from source is going to be dynamic-only and we're already putting it at the front of the PATH.

[vadorovsky]

Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @tamird)

---

Cargo.lock line 84 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

we now have two versions of llvm-sys, is that what we want?

sheesh, no, good catch

---

.github/workflows/ci.yml line 121 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

why?

Because otherwise, llvm-sys fails to build with this error:

https://github.com/aya-rs/bpf-linker/actions/runs/10328832485/job/28595877283#step:17:20

     Compiling llvm-sys v191.0.0-rc1
     Compiling bpf-linker v0.9.12 (/home/runner/work/bpf-linker/bpf-linker)
  error: could not find native static library `Polly`, perhaps an -L flag is missing?
  
  error: could not compile `llvm-sys` (lib) due to 1 previous error
  warning: build failed, waiting for other jobs to finish...
error: process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo build --manifest-path Cargo.toml --no-default-features` (exit status: 101)

I know that pulling libpolly is wasteful, but to not pull it, we would need to make build.rs in llvm-sys smarter. Currently, it just relies on the output ofllvm-config --system-libs:

https://gitlab.com/taricorp/llvm-sys.rs/-/blob/main/build.rs?ref_type=heads#L297

Perhaps that needs a proper fix in llvm-config itself, I need to dig in more.

---

.github/workflows/ci.yml line 153 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

just wondering: do we need to force it at all? can we just let it pick? the LLVM we build from source is going to be dynamic-only and we're already putting it at the front of the PATH.

Let me try.

[tamird]

Reviewed 3 of 3 files at r4, 2 of 2 files at r5, 1 of 1 files at r6, 2 of 2 files at r7, all commit messages.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @vadorovsky)

---

-- commits line 8 at r4:
Needs an update

---

.github/workflows/ci.yml line 121 at r1 (raw file):

Previously, vadorovsky (Michal Rostecki) wrote…

Because otherwise, llvm-sys fails to build with this error:

https://github.com/aya-rs/bpf-linker/actions/runs/10328832485/job/28595877283#step:17:20

     Compiling llvm-sys v191.0.0-rc1
     Compiling bpf-linker v0.9.12 (/home/runner/work/bpf-linker/bpf-linker)
  error: could not find native static library `Polly`, perhaps an -L flag is missing?
  
  error: could not compile `llvm-sys` (lib) due to 1 previous error
  warning: build failed, waiting for other jobs to finish...
error: process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo build --manifest-path Cargo.toml --no-default-features` (exit status: 101)

I know that pulling libpolly is wasteful, but to not pull it, we would need to make build.rs in llvm-sys smarter. Currently, it just relies on the output ofllvm-config --system-libs:

https://gitlab.com/taricorp/llvm-sys.rs/-/blob/main/build.rs?ref_type=heads#L297

Perhaps that needs a proper fix in llvm-config itself, I need to dig in more.

Include in commit message and maybe add a to-do?

---

.github/workflows/ci.yml line 153 at r1 (raw file):

Previously, vadorovsky (Michal Rostecki) wrote…

Let me try.

Needs an update in the commit message

[OmriSteiner]

I was able to build my project with an up to date Rust nightly using this branch.

[vadorovsky]

Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @tamird)

---

-- commits line 12 at r2:

Previously, tamird (Tamir Duberstein) wrote…

please include rationale

Done.

---

-- commits line 8 at r4:

Previously, tamird (Tamir Duberstein) wrote…

Needs an update

Done.

---

.github/workflows/ci.yml line 121 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

Include in commit message and maybe add a to-do?

Done.

---

.github/workflows/ci.yml line 153 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

Needs an update in the commit message

Done.

[tamird]

Reviewed 3 of 3 files at r8, 2 of 2 files at r9, 1 of 1 files at r10, 2 of 2 files at r11, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @vadorovsky)

---

-- commits line 7 at r8:
usage (typo)

---

.github/workflows/ci.yml line 122 at r8 (raw file):

          sudo apt update
          # TODO(vadorovsky): Remove the requirement of libpolly. It's being
          # unnecessarily required by build.rs of llvm-sys, which does so based

is it really required by build.rs? i see no mention of libpolly in https://gitlab.com/taricorp/llvm-sys.rs/-/blob/main/build.rs - what am I missing?

[tamird]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @vadorovsky)

---

-- commits line 37 at r11:
just squash this into the first commit?

[vadorovsky]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @tamird)

---

.github/workflows/ci.yml line 122 at r8 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

is it really required by build.rs? i see no mention of libpolly in https://gitlab.com/taricorp/llvm-sys.rs/-/blob/main/build.rs - what am I missing?

build.rs is calling llvm-config --libnames --link-[dynamic|static] here:

https://gitlab.com/taricorp/llvm-sys.rs/-/blob/48a97a9b4a33a25513e875ac0899fa0f8403418f/build.rs#L478

which returns:

llvm-config --libnames --link-static
libLLVMWindowsManifest.a libLLVMXRay.a libLLVMLibDriver.a libLLVMDlltoolDriver.a libLLVMTextAPIBinaryReader.a libLLVMCoverage.a libLLVMLineEditor.a libLLVMSandboxIR.a libLLVMXtensaDisassembler.a libLLVMXtensaAsmParser.a libLLVMXtensaCodeGen.a libLLVMXtensaDesc.a libLLVMXtensaInfo.a libLLVMM68kDisassembler.a libLLVMM68kAsmParser.a libLLVMM68kCode
Gen.a libLLVMM68kDesc.a libLLVMM68kInfo.a libLLVMXCoreDisassembler.a libLLVMXCoreCodeGen.a libLLVMXCoreDesc.a libLLVMXCoreInfo.a libLLVMX86TargetMCA.a libLLVMX86Disassembler.a libLLVMX86AsmParser.a libLLVMX86CodeGen.a libLLVMX86Desc.a libLLVMX86Info.a libLLVMWebAssemblyDisassembler.a libLLVMWebAssemblyAsmParser.a libLLVMWebAssemblyCodeGen.a libLLVMWebAssemblyUtils.a libLLVMWebAssemblyDesc.a libLLVMWebAssemblyInfo.a libLLVMVEDisassembler.a libLLVMVEAsmParser.a libLLVMVECodeGen.a libLLVMVEDesc.a libLLVMVEInfo.a libLLVMSystemZDisassembler.a libLLVMSystemZAsmParser.a libLLVMSystemZCodeGen.a libLLVMSystemZDesc.a libLLVMSystemZInfo.a libLLVMSparcDisassembler.a libLLVMSparcAsmParser.a libLLVMSparcCodeGen.a libLLVMSparcDesc.a libLLVMSparcInfo.a libLLVMRISCVTargetMCA.a libLLVMRISCVDisassembler.a libLLVMRISCVAsmParser.a libLLVMRISCVCodeGen.a libLLVMRISCVDesc.a libLLVMRISCVInfo.a libLLVMPowerPCDisassembler.a libLLVMPowerPCAsmParser.a libLLVMPowerPCCodeGen.a libLLVMPowerPCDesc.a libLLVMPowerPCInfo.a libLLVMNVPTXCodeGen.a libLLVMNVPTXDesc.a libLLVMNVPTXInfo.a libLLVMMSP430Disassembler.a libLLVMMSP430AsmParser.a libLLVMMSP430CodeGen.a libLLVMMSP430Desc.a libLLVMMSP430Info.a libLLVMMipsDisassembler.a libLLVMMipsAsmParser.a libLLVMMipsCodeGen.a libLLVMMipsDesc.a libLLVMMipsInfo.a libLLVMLoongArchDisassembler.a libLLVMLoongArchAsmParser.a libLLVMLoongArchCodeGen.a libLLVMLoongArchDesc.a libLLVMLoongArchInfo.a libLLVMLanaiDisassembler.a libLLVMLanaiCodeGen.a libLLVMLanaiAsmParser.a libLLVMLanaiDesc.a libLLVMLanaiInfo.a libLLVMHexagonDisassembler.a libLLVMHexagonCodeGen.a libLLVMHexagonAsmParser.a libLLVMHexagonDesc.a libLLVMHexagonInfo.a libLLVMBPFDisassembler.a libLLVMBPFAsmParser.a libLLVMBPFCodeGen.a libLLVMBPFDesc.a libLLVMBPFInfo.a libLLVMAVRDisassembler.a libLLVMAVRAsmParser.a libLLVMAVRCodeGen.a libLLVMAVRDesc.a libLLVMAVRInfo.a libLLVMARMDisassembler.a libLLVMARMAsmParser.a libLLVMARMCodeGen.a libLLVMARMDesc.a libLLVMARMUtils.a libLLVMARMInfo.a libLLVMAMDGPUTargetMCA.a libLLVMAMDGPUDisassembler.a libLLVMAMDGPUAsmParser.a libLLVMAMDGPUCodeGen.a libLLVMAMDGPUDesc.a libLLVMAMDGPUUtils.a libLLVMAMDGPUInfo.a libLLVMAArch64Disassembler.a libLLVMAArch64AsmParser.a libLLVMAArch64CodeGen.a libLLVMAArch64Desc.a libLLVMAArch64Utils.a libLLVMAArch64Info.a libLLVMPerfJITEvents.a libLLVMOrcDebugging.a libLLVMOrcJIT.a libLLVMWindowsDriver.a libLLVMMCJIT.a libLLVMJITLink.a libLLVMInterpreter.a libLLVMExecutionEngine.a libLLVMRuntimeDyld.a libLLVMOrcTargetProcess.a libLLVMOrcShared.a libLLVMDWP.a libLLVMDebugInfoLogicalView.a libLLVMDebugInfoGSYM.a libLLVMOption.a libLLVMObjectYAML.a libLLVMObjCopy.a libLLVMMCA.a libLLVMMCDisassembler.a libLLVMLTO.a libLLVMFrontendOpenACC.a libLLVMFrontendHLSL.a libLLVMFrontendDriver.a libLLVMExtensions.a libPolly.a libPollyISL.a libLLVMPasses.a libLLVMHipStdPar.a libLLVMCoroutines.a libLLVMCFGuard.a libLLVMipo.a libLLVMInstrumentation.a libLLVMVectorize.a libLLVMLinker.a libLLVMFrontendOpenMP.a libLLVMFrontendOffloading.a libLLVMDWARFLinkerParallel.a libLLVMDWARFLinkerClassic.a libLLVMDWARFLinker.a libLLVMCodeGenData.a libLLVMGlobalISel.a libLLVMMIRParser.a libLLVMAsmPrinter.a libLLVMSelectionDAG.a libLLVMCodeGen.a libLLVMTarget.a libLLVMObjCARCOpts.a libLLVMCodeGenTypes.a libLLVMIRPrinter.a libLLVMInterfaceStub.a libLLVMFileCheck.a libLLVMFuzzMutate.a libLLVMScalarOpts.a libLLVMInstCombine.a libLLVMAggressiveInstCombine.a libLLVMTransformUtils.a libLLVMBitWriter.a libLLVMAnalysis.a libLLVMProfileData.a libLLVMSymbolize.a libLLVMDebugInfoBTF.a libLLVMDebugInfoPDB.a libLLVMDebugInfoMSF.a libLLVMDebugInfoDWARF.a libLLVMObject.a libLLVMTextAPI.a libLLVMMCParser.a libLLVMIRReader.a libLLVMAsmParser.a libLLVMMC.a libLLVMDebugInfoCodeView.a libLLVMBitReader.a libLLVMFuzzerCLI.a libLLVMCore.a libLLVMRemarks.a libLLVMBitstreamReader.a libLLVMBinaryFormat.a libLLVMTargetParser.a libLLVMTableGen.a libLLVMSupport.a libLLVMDemangle.a

You can see libPolly.a and libPollyISL.a being mentioned there. And then build.rs just blindly follows and tries to link all these libs.

[vadorovsky]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @tamird)

---

.github/workflows/ci.yml line 122 at r8 (raw file):

Previously, vadorovsky (Michal Rostecki) wrote…

build.rs is calling llvm-config --libnames --link-[dynamic|static] here:

https://gitlab.com/taricorp/llvm-sys.rs/-/blob/48a97a9b4a33a25513e875ac0899fa0f8403418f/build.rs#L478

which returns:

llvm-config --libnames --link-static
libLLVMWindowsManifest.a libLLVMXRay.a libLLVMLibDriver.a libLLVMDlltoolDriver.a libLLVMTextAPIBinaryReader.a libLLVMCoverage.a libLLVMLineEditor.a libLLVMSandboxIR.a libLLVMXtensaDisassembler.a libLLVMXtensaAsmParser.a libLLVMXtensaCodeGen.a libLLVMXtensaDesc.a libLLVMXtensaInfo.a libLLVMM68kDisassembler.a libLLVMM68kAsmParser.a libLLVMM68kCode
Gen.a libLLVMM68kDesc.a libLLVMM68kInfo.a libLLVMXCoreDisassembler.a libLLVMXCoreCodeGen.a libLLVMXCoreDesc.a libLLVMXCoreInfo.a libLLVMX86TargetMCA.a libLLVMX86Disassembler.a libLLVMX86AsmParser.a libLLVMX86CodeGen.a libLLVMX86Desc.a libLLVMX86Info.a libLLVMWebAssemblyDisassembler.a libLLVMWebAssemblyAsmParser.a libLLVMWebAssemblyCodeGen.a libLLVMWebAssemblyUtils.a libLLVMWebAssemblyDesc.a libLLVMWebAssemblyInfo.a libLLVMVEDisassembler.a libLLVMVEAsmParser.a libLLVMVECodeGen.a libLLVMVEDesc.a libLLVMVEInfo.a libLLVMSystemZDisassembler.a libLLVMSystemZAsmParser.a libLLVMSystemZCodeGen.a libLLVMSystemZDesc.a libLLVMSystemZInfo.a libLLVMSparcDisassembler.a libLLVMSparcAsmParser.a libLLVMSparcCodeGen.a libLLVMSparcDesc.a libLLVMSparcInfo.a libLLVMRISCVTargetMCA.a libLLVMRISCVDisassembler.a libLLVMRISCVAsmParser.a libLLVMRISCVCodeGen.a libLLVMRISCVDesc.a libLLVMRISCVInfo.a libLLVMPowerPCDisassembler.a libLLVMPowerPCAsmParser.a libLLVMPowerPCCodeGen.a libLLVMPowerPCDesc.a libLLVMPowerPCInfo.a libLLVMNVPTXCodeGen.a libLLVMNVPTXDesc.a libLLVMNVPTXInfo.a libLLVMMSP430Disassembler.a libLLVMMSP430AsmParser.a libLLVMMSP430CodeGen.a libLLVMMSP430Desc.a libLLVMMSP430Info.a libLLVMMipsDisassembler.a libLLVMMipsAsmParser.a libLLVMMipsCodeGen.a libLLVMMipsDesc.a libLLVMMipsInfo.a libLLVMLoongArchDisassembler.a libLLVMLoongArchAsmParser.a libLLVMLoongArchCodeGen.a libLLVMLoongArchDesc.a libLLVMLoongArchInfo.a libLLVMLanaiDisassembler.a libLLVMLanaiCodeGen.a libLLVMLanaiAsmParser.a libLLVMLanaiDesc.a libLLVMLanaiInfo.a libLLVMHexagonDisassembler.a libLLVMHexagonCodeGen.a libLLVMHexagonAsmParser.a libLLVMHexagonDesc.a libLLVMHexagonInfo.a libLLVMBPFDisassembler.a libLLVMBPFAsmParser.a libLLVMBPFCodeGen.a libLLVMBPFDesc.a libLLVMBPFInfo.a libLLVMAVRDisassembler.a libLLVMAVRAsmParser.a libLLVMAVRCodeGen.a libLLVMAVRDesc.a libLLVMAVRInfo.a libLLVMARMDisassembler.a libLLVMARMAsmParser.a libLLVMARMCodeGen.a libLLVMARMDesc.a libLLVMARMUtils.a libLLVMARMInfo.a libLLVMAMDGPUTargetMCA.a libLLVMAMDGPUDisassembler.a libLLVMAMDGPUAsmParser.a libLLVMAMDGPUCodeGen.a libLLVMAMDGPUDesc.a libLLVMAMDGPUUtils.a libLLVMAMDGPUInfo.a libLLVMAArch64Disassembler.a libLLVMAArch64AsmParser.a libLLVMAArch64CodeGen.a libLLVMAArch64Desc.a libLLVMAArch64Utils.a libLLVMAArch64Info.a libLLVMPerfJITEvents.a libLLVMOrcDebugging.a libLLVMOrcJIT.a libLLVMWindowsDriver.a libLLVMMCJIT.a libLLVMJITLink.a libLLVMInterpreter.a libLLVMExecutionEngine.a libLLVMRuntimeDyld.a libLLVMOrcTargetProcess.a libLLVMOrcShared.a libLLVMDWP.a libLLVMDebugInfoLogicalView.a libLLVMDebugInfoGSYM.a libLLVMOption.a libLLVMObjectYAML.a libLLVMObjCopy.a libLLVMMCA.a libLLVMMCDisassembler.a libLLVMLTO.a libLLVMFrontendOpenACC.a libLLVMFrontendHLSL.a libLLVMFrontendDriver.a libLLVMExtensions.a libPolly.a libPollyISL.a libLLVMPasses.a libLLVMHipStdPar.a libLLVMCoroutines.a libLLVMCFGuard.a libLLVMipo.a libLLVMInstrumentation.a libLLVMVectorize.a libLLVMLinker.a libLLVMFrontendOpenMP.a libLLVMFrontendOffloading.a libLLVMDWARFLinkerParallel.a libLLVMDWARFLinkerClassic.a libLLVMDWARFLinker.a libLLVMCodeGenData.a libLLVMGlobalISel.a libLLVMMIRParser.a libLLVMAsmPrinter.a libLLVMSelectionDAG.a libLLVMCodeGen.a libLLVMTarget.a libLLVMObjCARCOpts.a libLLVMCodeGenTypes.a libLLVMIRPrinter.a libLLVMInterfaceStub.a libLLVMFileCheck.a libLLVMFuzzMutate.a libLLVMScalarOpts.a libLLVMInstCombine.a libLLVMAggressiveInstCombine.a libLLVMTransformUtils.a libLLVMBitWriter.a libLLVMAnalysis.a libLLVMProfileData.a libLLVMSymbolize.a libLLVMDebugInfoBTF.a libLLVMDebugInfoPDB.a libLLVMDebugInfoMSF.a libLLVMDebugInfoDWARF.a libLLVMObject.a libLLVMTextAPI.a libLLVMMCParser.a libLLVMIRReader.a libLLVMAsmParser.a libLLVMMC.a libLLVMDebugInfoCodeView.a libLLVMBitReader.a libLLVMFuzzerCLI.a libLLVMCore.a libLLVMRemarks.a libLLVMBitstreamReader.a libLLVMBinaryFormat.a libLLVMTargetParser.a libLLVMTableGen.a libLLVMSupport.a libLLVMDemangle.a

You can see libPolly.a and libPollyISL.a being mentioned there. And then build.rs just blindly follows and tries to link all these libs.

You can do the following to reproduce:

podman run --rm -it docker.io/ubuntu:22.04
apt update
apt install lsb-release wget software-properties-common gnupg
./llvm.sh 19 all
/usr/lib/llvm-19/bin/llvm-config --libnames --link-static

[vadorovsky]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @tamird)

---

.github/workflows/ci.yml line 122 at r8 (raw file):

Previously, vadorovsky (Michal Rostecki) wrote…

You can do the following to reproduce:

podman run --rm -it docker.io/ubuntu:22.04
apt update
apt install lsb-release wget software-properties-common gnupg
./llvm.sh 19 all
/usr/lib/llvm-19/bin/llvm-config --libnames --link-static

uhh, sorry, missed few commands

podman run --rm -it docker.io/ubuntu:22.04
apt update
apt install lsb-release wget software-properties-common gnupg
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
./llvm.sh 19 all
/usr/lib/llvm-19/bin/llvm-config --libnames --link-static

[tamird]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @vadorovsky)

---

.github/workflows/ci.yml line 122 at r8 (raw file):

Previously, vadorovsky (Michal Rostecki) wrote…

uhh, sorry, missed few commands

podman run --rm -it docker.io/ubuntu:22.04
apt update
apt install lsb-release wget software-properties-common gnupg
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
./llvm.sh 19 all
/usr/lib/llvm-19/bin/llvm-config --libnames --link-static

Got it. It's worthwhile to capture some of this here - the current phrasing suggests that build.rs is directly referencing libpolly.

FWIW this definitely looks like a packaging bug - it looks as if LLVM was built with libpolly but libpolly wasn't included in the package.

[vadorovsky]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @tamird)

---

.github/workflows/ci.yml line 122 at r8 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

Got it. It's worthwhile to capture some of this here - the current phrasing suggests that build.rs is directly referencing libpolly.

FWIW this definitely looks like a packaging bug - it looks as if LLVM was built with libpolly but libpolly wasn't included in the package.

Yeah, it looks like apt.llvm.org folks are just building the whole monorepo at once with powerset of cmake options and then putting different libs and binaries to different deb packages.

I quickly checked on Fedora and Gentoo and the problem doesn't exist there, each package (llvm(-dev), compiler-rt, polly(-dev)) is being built separately, even though the sources are the same.

And here is a funny thing. I checked how is it in the LLVM tarball from Rust. The output is empty 💀

$ ./bin/llvm-config --libnames --link-static

$

But it's fine, we can just define appropriate -L flags in RUSTFLAGS. I think I even prefer it this way. 😛

Pushed the change with more verbose comment and commit message.

[tamird]

Reviewed 3 of 3 files at r12, 1 of 1 files at r13, 1 of 1 files at r14, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @vadorovsky)