EventStream::~EventStream() causes dead-lock when exit() is called in alpaka kernel

[jkelling]

Some runtimes (Clang's OpenMP target offload) call exit() when an error occurs.

In picongpu (at least using a blocking queue) this causes a deadlock when the pmacc::EventStream destructor is called, which calls cuplaStreamSynchronize(), which uses the alpaka queues' wait functionality, which tries to lock the queues' mutex. The lock in this mutex is held by the enqueue function which never returned in this case.

I do know if this can be feasibly fixed, just want to bring it up for discussion.

[psychocoderHPC]

pmacc::EventStream should only be destructed at the end of the simulation. Do I am correct that this deadlock is showing up at the end of the simulation?

[jkelling]

That depends on how you define "end of simulation": It is not at the intended end of the simulation.

clang's OpenMP target offload runtime encounters an error in or around a target region (i.e. inside the alpaka TaskKernel*::operator() function. After printing an error message with zero information, it calls exit(), which ends the program right there.

I think it would be nicer of the runtime to raise an exception or trigger sigabrt (like assert() ), but at the moment it is not and it may not be the last runtime to decide to go this route.

[sbastrakov]

I thought std::exit is a variant of a normal end of the simulation (including returning from main()), with destructors properly called. Then I suppose the mutexes should be also unlocked by their destructors. Perhaps I have a wrong idea about it.

[jkelling]

No, I think you are right about exit.

The problem seems to be, that some constructors are called, but the stack is not unwound properly.

• The enqueue trait implementation of QueueGenericBlocking (formerly QueueCpuBlocking) creates a lock_guard before invoking the actual kernel.
• When exit() is called , the function do not return, but the call stack becomes deeper and it calls the pmacc::EventStream dtor before it might call the lock_guard dtor. You can test this for example by calling exit() here: https://github.com/ComputationalRadiationPhysics/picongpu/blob/dev/thirdParty/cupla/alpaka/include/alpaka/kernel/TaskKernelCpuSerial.hpp#L132 . If you run it in a debugger and hit ctrl+c when it is stuck you can get the backtrace.

[sbastrakov]

Ah, thanks for clarifications. So it seems we can't really force stack unwinding in this case, which feels bad.

[jkelling]

cpp-reference.com confirms, that std::exit does not unwind the stack. It talks about std::atexit, to register handlers, but using this does seem to work.

Apparently the atexit handler is called after the pmacc::EventStream dtor. (put that test in the wrong place, might work but would require the lock to be registered inside a singleton or otherwise global instance.)

[jkelling]

I filed an enhancement request with clang for this issue. I think the library should call abort() in case of a fatal error.
https://bugs.llvm.org/show_bug.cgi?id=46515