LLJIT::deinitialize not running static dtors in LLVM-17

[xybu]

So I have a program that highly resembles the LLJITWithInitializers.cpp example except that we expose all symbols in process memory to JIT space.

Recently we're upgrading from LLVM-15 to LLVM-17 and found that our unit tests involving static dtors are broken. It appears that

• static dtors in JITed module don't run anymore with LLJIT::deinitialize(JD)
• Something generated by JIT is attached to the real __cxa_atexit() which causes segfault during shutdown.

Static ctors seem fine.
I was looking through the commits in Orc but too many to find something relevant.

Then I searched __cxa_atexit in the repo and found this comment about LocalCXXRuntimeOverrides in ExecutionUtils.h which seems to describe exactly what's happening. With LLJIT most details are sealed and we only call the clean interface.

I tried to do CXXRuntimeoverrides.enable(llJIT_->getMainJITDylib(), *mangle_) but got Duplicate definition of symbol '__dso_handle' for now. Still looking.

Platform is Linux x86_64.

I'm running out of ideas so wonder if any expert is aware of some changes to LLJIT that require users to do something different / explicit to handle static dtors or atexit() gracefully. Any help is super appreciated. Thank you!

[llvmbot]

@llvm/issue-subscribers-orcjit

Author: Xiangyu Bu (xybu)

So I have a program that highly resembles the [LLJITWithInitializers.cpp](https://github.com/llvm/llvm-project/blob/main/llvm/examples/OrcV2Examples/LLJITWithInitializers/LLJITWithInitializers.cpp) example except that we expose all symbols in process memory to JIT space.

Recently we're upgrading from LLVM-15 to LLVM-17 and found that our unit tests involving static dtors are broken. It appears that

• static dtors in JITed module don't run anymore with LLJIT::deinitialize(JD)
• Something generated by JIT is attached to the real __cxa_atexit() which causes segfault during shutdown.

I was looking through the commits in Orc but too many to find something relevant.

Then I searched __cxa_atexit in the repo and found this comment about LocalCXXRuntimeOverrides in ExecutionUtils.h which seems to describe exactly what's happening. With LLJIT most details are sealed and we only call the clean interface.

I tried to do CXXRuntimeoverrides.enable(llJIT_->getMainJITDylib(), *mangle_) but got Duplicate definition of symbol '__dso_handle' for now. Still looking.

Platform is Linux x86_64.

I'm running out of ideas so wonder if any expert is aware of some changes to LLJIT that require users to do something different / explicit to handle static dtors or atexit() gracefully. Any help is super appreciated. Thank you!

[EugeneZelenko]

Could you please try 19 or main branch?

[xybu]

Could you please try 19 or main branch?

Unfortunately in my case this is a company prod environment where I cannot decide which LLVM version to use :(

[xybu]

I'm reading the code of GenericLLVMIRPlatformSupport which has some logic to deal with cxa_atexit() and dtors. Some search shows me it's orphaned, as well as setUpGenericLLVMIRPlatform().

There's indeed some difference re how setUpGenericLLVMIRPlatform is called

• https://github.com/llvm/llvm-project/blob/release/15.x/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp#L852
• https://github.com/llvm/llvm-project/blob/release/17.x/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp#L1049

I wonder if it's relevant

[xybu]

I think I found the commits

This one changed things around setUpGenericLLVMIRPlatform() and how process symbols are exposed
371cb1a
but got reverted due to failures.

This commit then fixed it and the examples.
231107a

I suspect I can fix my issue the same way.

[xybu]

Confirmed changes like 231107a would fix the issue.

[xybu]

I wonder if communications regarding the commits can be done better so that we don't need to dive deep into code to figure out what happened. Did I miss something in the release notes maybe?

[lhames]

@xybu I'm afraid not -- I'm swamped, and often don't have the bandwidth to write release notes for the JIT. Any contributions to improving the documentation would be most appreciated!

[xybu]

I think one or two sentence like

"LLVM-17 brings a new and simpler way to expose process symbols to JIT space and old approach may not work anymore. Please follow 231107a to update your code."

would be great and can quickly point people to solution.

But I don't know how to contribute to release notes.