[lldb] Possible design issue when using Python >= 3.12

[tuliom]

While trying to update lldb to work with Python >= 3.12, I found the following issue:

Starting with Python 3.12, there is code now preventing the usage of PyImport_AppendInittab after Py_Initialize().
However, according to this source code comment:

We cannot control what external code may have done before getting to this point in LLDB,
including potentially having already initialized Python...

This is causing the following failures:

Failed Tests (19):
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/0/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/1/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/10/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/11/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/12/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/13/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/14/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/15/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/16/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/17/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/18/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/2/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/3/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/4/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/5/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/6/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/7/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/8/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/9/20

With the following output:

Fatal Python error: PyImport_AppendInittab: PyImport_AppendInittab() may not be called after Py_Initialize()
Python runtime state: initialized

Current thread 0x00007f60b876c300 (most recent call first):
  <no Python frame>
 #0 0x00007f60b059bceb llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/build/llvm-build.f39/lib/libLLVM-
18git.so+0x79bceb)
 #1 0x00007f60b05992ab SignalHandler(int) Signals.cpp:0:0
 #2 0x00007f60afa089a0 __restore_rt (/lib64/libc.so.6+0x3e9a0)
 #3 0x00007f60afa5a834 __pthread_kill_implementation (/lib64/libc.so.6+0x90834)
 #4 0x00007f60afa088ee gsignal (/lib64/libc.so.6+0x3e8ee)
 #5 0x00007f60af9f08ff abort (/lib64/libc.so.6+0x268ff)
 #6 0x00007f60b435acaa /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Python/pylifecycle.c:2712:0
 #7 0x00007f60b435acaa /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Python/pylifecycle.c:2820:0
 #8 0x00007f60b435b020 /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Modules/faulthandler.c:1035:0
 #9 0x00007f60b42b2a6c /usr/include/bits/string_fortified.h:59:10
#10 0x00007f60b42b2a6c PyImport_AppendInittab /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Python/import.c:1
502:5
#11 0x0000000000480faf lldb_private::ScriptInterpreterPythonImpl::Initialize() (/build/llvm-build.f39/tools/ll
db/unittests/ScriptInterpreter/Python/./ScriptInterpreterPythonTests+0x480faf)
#12 0x000000000044cded PythonTestSuite::SetUp() (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/
Python/./ScriptInterpreterPythonTests+0x44cded)
#13 0x0000000000441789 PythonDataObjectsTest::SetUp() NameMatches.cpp:0:0
#14 0x0000000000465687 testing::Test::Run() (.part.0) gtest-all.cc:0:0
#15 0x000000000046a58a testing::TestInfo::Run() (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/
Python/./ScriptInterpreterPythonTests+0x46a58a)
#16 0x000000000046abcc testing::TestSuite::Run() (.part.0) gtest-all.cc:0:0
#17 0x0000000000471d10 testing::internal::UnitTestImpl::RunAllTests() (/build/llvm-build.f39/tools/lldb/unitte
sts/ScriptInterpreter/Python/./ScriptInterpreterPythonTests+0x471d10)
#18 0x000000000046503e testing::UnitTest::Run() (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/
Python/./ScriptInterpreterPythonTests+0x46503e)
#19 0x000000000042ab26 main (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/Python/./ScriptInter
preterPythonTests+0x42ab26)
#20 0x00007f60af9f214a __libc_start_call_main (/lib64/libc.so.6+0x2814a)
#21 0x00007f60af9f220b __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2820b)
#22 0x000000000042b1a5 _start (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/Python/./ScriptInt
erpreterPythonTests+0x42b1a5)

One can reduce the list of failures to 2 by guaranteeing the critical code in InitializePythonRAII() is executed only once.
But it still doesn't solve all the cases.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Tulio Magno Quites Machado Filho (tuliom)

While trying to update lldb to work with Python >= 3.12, I found the following issue:

Starting with Python 3.12, there is code now preventing the usage of PyImport_AppendInittab after Py_Initialize().
However, according to this source code comment:

We cannot control what external code may have done before getting to this point in LLDB,
including potentially having already initialized Python...

This is causing the following failures:

Failed Tests (19):
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/0/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/1/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/10/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/11/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/12/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/13/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/14/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/15/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/16/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/17/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/18/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/2/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/3/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/4/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/5/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/6/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/7/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/8/20
  lldb-unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/9/20

With the following output:

Fatal Python error: PyImport_AppendInittab: PyImport_AppendInittab() may not be called after Py_Initialize()
Python runtime state: initialized

Current thread 0x00007f60b876c300 (most recent call first):
  <no Python frame>
 #<!-- -->0 0x00007f60b059bceb llvm::sys::PrintStackTrace(llvm::raw_ostream&amp;, int) (/build/llvm-build.f39/lib/libLLVM-
18git.so+0x79bceb)
 #<!-- -->1 0x00007f60b05992ab SignalHandler(int) Signals.cpp:0:0
 #<!-- -->2 0x00007f60afa089a0 __restore_rt (/lib64/libc.so.6+0x3e9a0)
 #<!-- -->3 0x00007f60afa5a834 __pthread_kill_implementation (/lib64/libc.so.6+0x90834)
 #<!-- -->4 0x00007f60afa088ee gsignal (/lib64/libc.so.6+0x3e8ee)
 #<!-- -->5 0x00007f60af9f08ff abort (/lib64/libc.so.6+0x268ff)
 #<!-- -->6 0x00007f60b435acaa /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Python/pylifecycle.c:2712:0
 #<!-- -->7 0x00007f60b435acaa /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Python/pylifecycle.c:2820:0
 #<!-- -->8 0x00007f60b435b020 /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Modules/faulthandler.c:1035:0
 #<!-- -->9 0x00007f60b42b2a6c /usr/include/bits/string_fortified.h:59:10
#<!-- -->10 0x00007f60b42b2a6c PyImport_AppendInittab /usr/src/debug/python3.12-3.12.0-1.fc39.x86_64/Python/import.c:1
502:5
#<!-- -->11 0x0000000000480faf lldb_private::ScriptInterpreterPythonImpl::Initialize() (/build/llvm-build.f39/tools/ll
db/unittests/ScriptInterpreter/Python/./ScriptInterpreterPythonTests+0x480faf)
#<!-- -->12 0x000000000044cded PythonTestSuite::SetUp() (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/
Python/./ScriptInterpreterPythonTests+0x44cded)
#<!-- -->13 0x0000000000441789 PythonDataObjectsTest::SetUp() NameMatches.cpp:0:0
#<!-- -->14 0x0000000000465687 testing::Test::Run() (.part.0) gtest-all.cc:0:0
#<!-- -->15 0x000000000046a58a testing::TestInfo::Run() (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/
Python/./ScriptInterpreterPythonTests+0x46a58a)
#<!-- -->16 0x000000000046abcc testing::TestSuite::Run() (.part.0) gtest-all.cc:0:0
#<!-- -->17 0x0000000000471d10 testing::internal::UnitTestImpl::RunAllTests() (/build/llvm-build.f39/tools/lldb/unitte
sts/ScriptInterpreter/Python/./ScriptInterpreterPythonTests+0x471d10)
#<!-- -->18 0x000000000046503e testing::UnitTest::Run() (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/
Python/./ScriptInterpreterPythonTests+0x46503e)
#<!-- -->19 0x000000000042ab26 main (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/Python/./ScriptInter
preterPythonTests+0x42ab26)
#<!-- -->20 0x00007f60af9f214a __libc_start_call_main (/lib64/libc.so.6+0x2814a)
#<!-- -->21 0x00007f60af9f220b __libc_start_main@<!-- -->GLIBC_2.2.5 (/lib64/libc.so.6+0x2820b)
#<!-- -->22 0x000000000042b1a5 _start (/build/llvm-build.f39/tools/lldb/unittests/ScriptInterpreter/Python/./ScriptInt
erpreterPythonTests+0x42b1a5)

One can reduce the list of failures to 2 by guaranteeing the critical code in InitializePythonRAII() is executed only once.
But it still doesn't solve all the cases.

[bulbazord]

@tuliom How far did you get here? I know you were able to do some work with LLDB's Python usage, is the failure list still at those 19?

I assume we (the LLDB maintainers) will need to contend with this at some point in the near future, so thank you for opening this issue.

[JDevlieghere]

FWIW I also just hit this with Python 3.12.

[tuliom]

@tuliom How far did you get here? I know you were able to do some work with LLDB's Python usage, is the failure list still at those 19?

@bulbazord I didn't start working on this issue. The 19 failures are remaining failures after my recent changes. I'm afraid that a larger re-design of the plugins mechanism is required in order to solve this issue.

[bulbazord]

@tuliom How far did you get here? I know you were able to do some work with LLDB's Python usage, is the failure list still at those 19?

@bulbazord I didn't start working on this issue. The 19 failures are remaining failures after my recent changes. I'm afraid that a larger re-design of the plugins mechanism is required in order to solve this issue.

Gotcha. Redesigning the python plugins is no small task but it will have to be done eventually. Thanks for bringing this to our attention.

[torokati44]

On Fedora 39, with the python3-lldb package installed (Python 3.12.0, LLVM 17.0.4), all one has to do to get this crash is:

$ python3
>>> import lldb

This leaves the Scripting Bridge API essentially unusable.

Of course, OP might be familiar with this, being the packager of said Fedora package... :)

EDIT: One workaround is to install the python3.11 package, and start the interpreter like this: PYTHONPATH=$(lldb -P) python3.11 [source].

[ferdnyc]

@tuliom

While trying to update lldb to work with Python >= 3.12, I found the following issue:
Starting with Python 3.12, there is code now preventing the usage of PyImport_AppendInittab after Py_Initialize().

Which sounds like a whole big mess, except for this one tantalizing note in the Python C API docs on initialization

If Python is initialized multiple times, PyImport_AppendInittab() or PyImport_ExtendInittab() must be called before each Python initialization.

While lldb can't control what other code has done before running python, including possibly initializing it, perhaps it's possible to reinitialize it, and therefore have access to PyImport_AppendInittab() again.

Presumably that would only be possible when embedding Python into LLDB using the C API, where the initialization can be adjusted ahead-of-time. Whereas the extension module might be better off avoiding any attempts to meddle with the Python initialization.

After all, what if the environment is an already-initialized, standalone Python interpreter that's only now loading the extension? Like in @torokati44's example of typing import lldb at the REPL prompt. Not only is Python readline already loaded, there, but it's active and managing interactive state for the user. Swapping it out in that context sounds like a Bad Time™.

cc: @bulbazord