SIGSEGV in phtread_exit after switching from gnustl_static to libc++_static

[andreya108]

Description

We have a project which is for a long time compiled gnustl_static in c++ code.

Jni tasks are running async in separate thread like this (simplified for readability):

new Thread(this::asyncRunnable).start()
...
void asyncRunnable() {
try { 
   callJni(); 
} catch () {
...
} finally {
...
}
// everything is ok
} // <- sigsegv here

So, everything is fine when all code is linked with gnustl_static, but when I've tried libc++_static then I got SIGSEGV when finishing java thread (actual stacktrace points to releasing threadlocal data in pthread_exit).

Can anyone please help me figuring out what is wrong with libc++?

Environment Details

• NDK Version: r14b
• Build sytem: ndk-build
• Host OS: Ubuntu 16.04
• Compiler: Clang -std=c++14
• ABI: arm64-v8a
• STL: libc++
• NDK API level: 21
• Device API level: 25

[enh]

do you have a reproduceable example we can actually build?

a stack trace of a real crash?

[andreya108]

Here is stack trace:

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x7c41fc040c
Stack frame #00 pc 0000007c41fc040c  <unknown>
Stack frame #01 pc 0000000000068d44  /system/lib64/libc.so (_Z21pthread_key_clean_allv+116)
Stack frame #02 pc 0000000000068874  /system/lib64/libc.so (pthread_exit+68)
Stack frame #03 pc 0000000000068738  /system/lib64/libc.so (_ZL15__pthread_startPv+212)
Stack frame #04 pc 000000000001da7c  /system/lib64/libc.so (__start_thread+16)

Unfortunately I have no reproducable example yet, there are tons of proprietary code with many dependencies (like boost libs).. The only definite fact that it works normally with gnustl.

Another thing may be important, inside jni call the rest of code is loaded on-demand by dlopen (libcore.so). So we have a pair of dlopen/dlclose inside the call. (preventing keeping in memory huge lib when it is not actually needed)

And std::string& argument is passing through libjni.so -> libcore.so

[andreya108]

Here is complete stack trace (whithout ndk-trace shortening):

04-10 18:06:30.952 16905-16905/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
04-10 18:06:30.953 16905-16905/? A/DEBUG: Build fingerprint: 'google/bullhead/bullhead:7.1.2/NPG05D/3635581:user/release-keys'
04-10 18:06:30.953 16905-16905/? A/DEBUG: Revision: 'rev_1.0'
04-10 18:06:30.953 16905-16905/? A/DEBUG: ABI: 'arm64'
04-10 18:06:30.953 16905-16905/? A/DEBUG: pid: 16098, tid: 16365, name: X  >>> com.x.y <<<
04-10 18:06:30.953 16905-16905/? A/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x7c41fc040c
04-10 18:06:30.958 16905-16905/? A/DEBUG: Abort message: ' 
                                          ==========Log Level 5 ========
                                          '
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x0   0000007c415b1cf0  x1   0000000000000000  x2   0000007c4702e380  x3   0000000000000000
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x4   000000000000002e  x5   0000000000000000  x6   0000000000000000  x7   0000000000000001
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x8   0000007c4272e5f0  x9   0000000000000001  x10  0000007c41fc040c  x11  0000000000000001
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x12  0000007c4272da18  x13  0000000000000007  x14  00000000ffffffff  x15  2e8ba2e8ba2e8ba3
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x16  0000007c62932458  x17  0000007c628da464  x18  0000000000ffffeb  x19  0000007c4272e450
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x20  0000000000000004  x21  0000007c62939988  x22  0000000000000002  x23  00000000000001a0
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x24  0000007c4272e4d0  x25  0000000000105000  x26  725df47256723079  x27  0000007c61625854
04-10 18:06:30.958 16905-16905/? A/DEBUG:     x28  0000007c617fe170  x29  0000007c4272e390  x30  0000007c628d9d48
04-10 18:06:30.958 16905-16905/? A/DEBUG:     sp   0000007c4272e360  pc   0000007c41fc040c  pstate 0000000060000000
04-10 18:06:30.966 16905-16905/? A/DEBUG: backtrace:
04-10 18:06:30.966 16905-16905/? A/DEBUG:     #00 pc 0000007c41fc040c  <unknown>
04-10 18:06:30.966 16905-16905/? A/DEBUG:     #01 pc 0000000000068d44  /system/lib64/libc.so (_Z21pthread_key_clean_allv+116)
04-10 18:06:30.966 16905-16905/? A/DEBUG:     #02 pc 0000000000068874  /system/lib64/libc.so (pthread_exit+68)
04-10 18:06:30.966 16905-16905/? A/DEBUG:     #03 pc 0000000000068738  /system/lib64/libc.so (_ZL15__pthread_startPv+212)
04-10 18:06:30.966 16905-16905/? A/DEBUG:     #04 pc 000000000001da7c  /system/lib64/libc.so (__start_thread+16)

[andreya108]

What is the meaning of 'Abort message' here? Looks like it's a string from our logging engine...

[DanAlbert]

And std::string& argument is passing through libjni.so -> libcore.so

Both libjni and libcore are your libraries, right? You don't mean libjavacore.so from the system? If you're passing STL objects between your code and the system you're gonna have a bad time since there isn't a stable C++ ABI for the platform.

[andreya108]

Sorry for unclear naming.

Yes, libjni and libcore have nothing to system libraries. And passing STL objects only inside my code, not to the system.

I'll try to make an example project when I have time...

[DanAlbert]

Thanks, I'll take a look once we have a test case. Unfortunately not much I can do until then.

[andreya108]

Here is a test case:
https://drive.google.com/open?id=0By1bFhqM0X9PYWlXVUNEUjEzbjA

I've played some time to get the problem within the above test case, but failed to reproduce all the same.

But I've faced very similar problem concerning both libc++ and gnustl.

When I introduced thread_local object, got the same behavior as I described originally.

But original code does not contain any explicit thread_locals (though thread locals might be there since the code is multithreaded and pthread is widely used in boost, mutexes, lockers, semaphores, etc..).

[thomasjammet]

I am facing the same problems with thread_local variables with both libc++ and gnustl (ndk r15).
It seems that threads and thread_local objects are deleted too late (after static objects destruction?).

[andreya108]

Unfortunately, it breaks even C++11 support (when thread_local appeared), currently in the middle of 2017...

Is everything clear with the issue and you (NDK Developers) can reproduce it?

[DanAlbert]

Sorry, haven't had a chance to look back at this yet, but yes, now that we have source there should be something actionable here. I'll let you know if for some reason it doesn't repro.