test_gdb failed with 15 min timeout on PPC64LE RHEL8 3.x (LTO + pydebug)

[vstinner]

PPC64LE RHEL8 3.x:

0:19:19 load avg: 1.37 [463/463/1] test_gdb process crashed (Exit code 1)
Timeout (0:15:00)!
Thread 0x00007fffae664d70 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 398 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2108 in _communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1209 in communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 107 in run_gdb
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 224 in get_stack_trace
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 949 in test_pycfunction
  (...)

(...)

0:19:19 load avg: 1.37 Re-running 1 failed tests in verbose mode in subprocesses
0:19:19 load avg: 1.37 Run 1 test in parallel using 1 worker process (timeout: 15 min, worker timeout: 20 min)
0:19:49 load avg: 1.22 running (1): test_gdb (30.0 sec)
0:20:19 load avg: 1.19 running (1): test_gdb (1 min)
0:20:49 load avg: 1.11 running (1): test_gdb (1 min 30 sec)
(...)
0:33:50 load avg: 1.09 running (1): test_gdb (14 min 30 sec)
0:34:20 load avg: 1.14 running (1): test_gdb (15 min)
0:34:20 load avg: 1.14 [1/1/1] test_gdb process crashed (Exit code 1)
Re-running test_gdb in verbose mode
test_NULL_ob_type (test.test_gdb.PrettyPrintTests.test_NULL_ob_type)
Ensure that a PyObject* with NULL ob_type is handled gracefully ... ok
(...)
test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction)
Verify that "py-bt" displays invocations of PyCFunction instances ... Timeout (0:15:00)!
Thread 0x00007fffa14b4d70 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 398 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2108 in _communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1209 in communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 107 in run_gdb
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 224 in get_stack_trace
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 949 in test_pycfunction
  (...)

build: https://buildbot.python.org/all/#/builders/559/builds/3951

Linked PRs

• gh-109972: Split test_gdb into a package of 3 tests #109977
• gh-109972: Enhance test_gdb #110026
• [3.12] gh-109972: Split test_gdb.py into test_gdb package (#109977) #110339
• [3.11] gh-109972: Split test_gdb.py into test_gdb package (#109977) (… #110343
• [3.12] gh-109972: Enhance test_gdb (#110026) #110351
• [3.11] [3.12] gh-109972: Enhance test_gdb (GH-110026) (GH-110351) #110354

[vstinner]

Ah, it's a LTO build.

configure + make:

../configure --prefix '$(PWD)/target' --with-pydebug --with-platlibdir=lib64 --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb --enable-loadable-sqlite-extensions --with-ssl-default-suites=openssl --without-static-libpython --with-lto
(...)
gcc -pthread -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include   -fPIC -DPy_BUILD_CORE -o Programs/python.o ../Programs/python.c
(...)
gcc -pthread   -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g  -Xlinker -export-dynamic -o python Programs/python.o -L. -lpython3.13d -lpthread -ldl  -lutil                        -lm 

test.pythoninfo:

libregrtests.build_info: debug shared LTO

CC.version: gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)

sysconfig[PY_CFLAGS]: -fno-strict-overflow -Wsign-compare -g -Og -Wall
sysconfig[PY_CFLAGS_NODIST]: -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../Include/internal
sysconfig[PY_STDMODULE_CFLAGS]: -fno-strict-overflow -Wsign-compare -g -Og -Wall -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -fPIC

sysconfig[PY_LDFLAGS_NODIST]: -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g
sysconfig[PY_CORE_LDFLAGS]: -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g

[vstinner]

On this buildbot, if Python is built with ./configure --with-pydebug and gcc -Og, test_gdb completes in 4 min 30 sec and looks reliable:

[vstinner@rh-power-vm13 cpython]$ ./python -m test test_gdb -u all -v
(...)
Total duration: 4 min 30 sec
Total tests: run=46 skipped=1
Total test files: run=1/1
Result: SUCCESS

[vstinner]

When Python is built with ./configure --with-pydebug --with-lto && make -j10, well, test_gdb is ... way slower: 19 min 45 sec.

test_gdb passed in 19 min 45 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 19 min 45 sec
Total tests: run=46 skipped=1
Total test files: run=1/1
Result: SUCCESS

[vstinner]

Python built with ./configure --with-pydebug CFLAGS="-O0" and gcc -O0:

test_gdb passed in 4 min 12 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 4 min 12 sec
Total tests: run=46 skipped=1
Total test files: run=1/1
Result: SUCCESS

[vstinner]

The test runs 130 gdb commands in total. Average:

• gcc -O0: 1.9 sec / command
• gcc -Og: 2.1 sec / command
• gcc -Og -flto: 9.1 sec / command

[vstinner]

Number of gdb commands per test case:

PrettyPrintTests: 62

PyBtTests: 58

PyListTests: 6
PyLocalsTests: 5
PyPrintTests: 7
StackNavigationTests: 7

[vstinner]

I wrote PR #109977 to split test_gdb into a package of 3 tests so these 3 tests can be run in parallel.