Using LLVM (Clang++) to compile PALISADE for BPF target

[OluAgunloye]

@jackcmay

Hello,

This is a continuation of the issues I'm experiencing with PALISADE:

`PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0. Program arguments: /usr/bin/clang -target bpf -DMATHBACKEND=2 -DPARALLEL -I/root/palisade-development/third-party/include -I/root/palisade-development/third-party/cereal/include -I/root/palisade-development/third-party/google-test/googletest -I/root/palisade-development/third-party/google-test/googletest/include -I/root/palisade-development/src/core/include -I/root/palisade-development/build/src/core -I/root/palisade-development/src/core/lib -I/usr/include/c++/10 -I/root/include/x86_64-linux-gnu/c++/10 -D__x86_64__ -D__LP64__ -I/root/include/x86_64-linux-gnu -Wall -Werror -O3 -DPALISADE_VERSION=1.11.4 -Wno-unused-private-field -Wno-shift-op-parentheses -O3 -DNDEBUG -fPIC -std=gnu++11 -o CMakeFiles/coreobj.dir/lib/encoding/coefpackedencoding.cpp.o -c /root/palisade-development/src/core/lib/encoding/coefpackedencoding.cpp

1. parser at end of file
2. Code generation
3. Running pass 'Function Pass Manager' on module '/root/palisade-development/src/core/lib/encoding/coefpackedencoding.cpp'.
4. Running pass 'Live Variable Analysis' on function '@_ZN8lbcrypto18CoefPackedEncoding6EncodeEv'
   #0 0x00007f68cfa98ae3 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xbd9ae3)
   [LLD][BPF] Enable BPF shared object creation #1 0x00007f68cfa96df0 llvm::sys::RunSignalHandlers() (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xbd7df0)
   Fix Github Actions #2 0x00007f68cf9e7e76 (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xb28e76)
   [BPF] Disable llvm tests incompatible with Solana BPF backend #3 0x00007f68d730a1f0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x141f0)
   [BPF] Allow selectively disable compiler builtins for BPF target #4 0x00007f68cfd43e6c llvm::LiveVariables::runOnBlock(llvm::MachineBasicBlock*, unsigned int) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xe84e6c)
   [BPF] Map signed division operation to corresponding intrinsic function #5 0x00007f68cfd443cf llvm::LiveVariables::runOnMachineFunction(llvm::MachineFunction&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xe853cf)
   [SOL][BPF] Adjust BPF tests #6 0x00007f68cfda551e llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xee651e)
   [SOL][BPF] Allow misaligned loads #7 0x00007f68cfbc058d llvm::FPPassManager::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xd0158d)
   [SOL][BPF] Add BPF compiler-rt builtins #8 0x00007f68cfbc5f73 llvm::FPPassManager::runOnModule(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xd06f73)
   [SOL][BPF] Enable Solana BPF extensions as subtarget feature #9 0x00007f68cfbc0bdf llvm::legacy::PassManagerImpl::run(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xd01bdf)
   Fixes required for Solang #10 0x00007f68d5bf26f6 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_deletellvm::raw_pwrite_stream >) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x15446f6)
   [SOL][BPF] Remove debug printf #11 0x00007f68d5e8b8af (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x17dd8af)
   [SOL][BPF] Disable debug info when solana feature flag is set #12 0x00007f68d4fe6b64 clang::ParseAST(clang::Sema&, bool, bool) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x938b64)
   Upgrade to rust 1.52.1 #13 0x00007f68d657fd48 clang::FrontendAction::Execute() (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1ed1d48)
   [SOL] Set max stores per mem func depending on the target features #14 0x00007f68d650d7d1 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1e5f7d1)
   Add type attributes to LLVM C API #15 0x00007f68d65e2d62 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1f34d62)
   Using LLVM (Clang++) to compile PALISADE for BPF target #16 0x00000000004127e2 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/usr/bin/clang+0x4127e2)
   [SOL] Prevent breaking ISelDAG connectivity on replacing loads by con… #17 0x0000000000410b5e (/usr/bin/clang+0x410b5e)
   [SOL] Revert to R_BPF_64_32 until support for R_BPF_64_ABS32 added #18 0x00007f68d62296c2 (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b7b6c2)
   [SOL] Add sbf-solana-solana target triplet #19 0x00007f68cf9e7c4d llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/lib/x86_64-linux-gnu/libLLVM-12.so.1+0xb28c4d)
   [SOL] Turn on solana feature for SBF target by default #20 0x00007f68d6228eb9 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optionalllvm::StringRef >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool) const (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b7aeb9)
   [SOL] Register SBF asm parser #21 0x00007f68d61fe46f clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b5046f)
   [SOL] Add missing SBF conditions to match BPFEL target #22 0x00007f68d61fe827 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) const (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b50827)
   [SOL] add support for (pseudo) atomics to SBF #23 0x00007f68d621320c clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) (/lib/x86_64-linux-gnu/libclang-cpp.so.12+0x1b6520c)
   [SOL] disable llvm.bpf.load.* intrinsics on SBF #24 0x0000000000410434 main (/usr/bin/clang+0x410434)
   [SOL] re-enable debug info on SBF #25 0x00007f68ce977565 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x28565)
   Dynamic stack frame size support #26 0x000000000040dd1e _start (/usr/bin/clang+0x40dd1e)
   clang: error: clang frontend command failed with exit code 139 (use -v to see invocation)
   Ubuntu clang version 12.0.0-3ubuntu1~21.04.2
   Target: bpf
   Thread model: posix
   InstalledDir: /usr/bin
   clang: note: diagnostic msg:

---

​
PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /tmp/coefpackedencoding-e3e0eb.cpp
clang: note: diagnostic msg: /tmp/coefpackedencoding-e3e0eb.sh
clang: note: diagnostic msg:
​`

[OluAgunloye]

@jackcmay This seems almost exclusively a Clang++ issue too. Is there a g++ to BPF target for compiling LLVM?

[OluAgunloye]

Stuck here, @jackcmay . Streams seem to be a large issue (all types).

In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/complex:242: In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/sstream:184: In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/istream:163: In file included from /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/ostream:140: /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:750:26: error: use of undeclared identifier 'strtoll_l' long long __ll = strtoll_l(__a, &__p2, __base, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:790:35: error: use of undeclared identifier 'strtoull_l' unsigned long long __ll = strtoull_l(__a, &__p2, __base, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:819:12: error: use of undeclared identifier 'strtof_l' return strtof_l(__a, __p2, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:825:12: error: use of undeclared identifier 'strtod_l' return strtod_l(__a, __p2, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:831:12: error: use of undeclared identifier 'strtold_l' return strtold_l(__a, __p2, _LIBCPP_GET_C_LOCALE); ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/locale:1105:9: error: use of undeclared identifier 'sscanf_l' if (__libcpp_sscanf_l(__buf.c_str(), _LIBCPP_GET_C_LOCALE, "%p", &__v) != 1) ^ /home/alien/Desktop/Code/llvm-project/build/include/c++/v1/__bsd_locale_defaults.h:34:61: note: expanded from macro '__libcpp_sscanf_l' #define __libcpp_sscanf_l(...) sscanf_l(__VA_ARGS__)

[msergey08]

@jackcmay can clang compile any std c++ project with solana bpf-tools on host target x86_64-linux-gnu?
where can we get bpf stdc++ headers and libraries?

[msergey08]

@jackcmay can solana-labs:solana-rustc branch solve this problem?

[jackcmay]

We don't provide any bpf compiled standard libs for C/C++ compilation yet. In order to support that the libraries would need to be modified to remove all system calls as well as probably a bunch of other changes to get it to build. We have been hesitant to pick a standard library since the C space is so fragmented. Picking glibc might be ok. @cpp-guru do you have a recommendation? @dmakarov thoughts on us releasing a bpf/solana compatible stdlib?

[dmakarov]

We may try compiling newlib for BPF configuring syscalls so that they don't conflict with our runtime restrictions. This will not help building PALISADE, however.

As an experiment I can probably try compiling llvm-project/libcxx for BPF, but I'm afraid it's going to be a substantial struggle to have it built. I guess if we want C/C++ toolchain to be useful for on-chain program development, we need to provide the standard libraries, eventually.

[msergey08]

@jackcmay "Picking glibc might be ok." - yes. However, I'd like Solana dev team to provide standard C++ libraries as well. Why don't we use C++ libraries for Solana on-chain program? Many C++ libraries(for example, Palisade) will be waiting to be easily implemented on chain.

[OluAgunloye]

@jackcmay @cpp-guru @dmakarov
Can we work together on this?

Would like to help get this done for obvious personal reasons, but also because it's about time this is done anyway. Would be willing to put up some funds to make it work.

[jackcmay]

@OluAgunloye Thanks for the interest! @dmakarov and I will sync up and get back to you.

[dmakarov]

@OluAgunloye as first step we're going to provide newlib (an implementation of C standard library) binaries compiled to BPF. Do you have any concerns about newlib?

[OluAgunloye]

@dmakarov
This potentially addresses the sys call issues, and I've taken a look at newlib myself and it seems quite versatile with its ability to resolve sys calls almost regardless of underlying system. It makes a lot of sense to try to recompile w/ BPF. But what to do about these C++ std libs...

[dmakarov]

@dmakarov
This potentially addresses the sys call issues, and I've taken a look at newlib myself and it seems quite versatile its sys call compatibility. It makes a lot of sense to try to recompile w/ BPF. But what to do about these C++ std libs...

We'll see if we can adapt llvm-project/libcxx (or part of it) to be buildable for BPF.

[OluAgunloye]

@dmakarov
This potentially addresses the sys call issues, and I've taken a look at newlib myself and it seems quite versatile its sys call compatibility. It makes a lot of sense to try to recompile w/ BPF. But what to do about these C++ std libs...

We'll see if we can adapt llvm-project/libcxx (or part of it) to be buildable for BPF.

Hello! Just thought I'd stop by to see how things are going. Does newlib compile easily enough to BPF?