`test_super_deep()` for `test_call` triggers a stack overflow protection under a debug build for WASI

[brettcannon]

Tested using WASI-SDK 20 and wasmtime 14.

Linked PRs

• gh-111798: Use lower Py_C_RECURSION_LIMIT in debug mode #112124
• GH-111798: skip test_super_deep() from test_call under pydebug builds on WASI #114010

[brettcannon]

cpython/Lib/test/test_call.py

Lines 992 to 1026 in 804575b

	@skip_on_s390x
	def test_super_deep(self):
	def recurse(n):
	if n:
	recurse(n-1)
	def py_recurse(n, m):
	if n:
	py_recurse(n-1, m)
	else:
	c_py_recurse(m-1)
	def c_recurse(n):
	if n:
	_testcapi.pyobject_vectorcall(c_recurse, (n-1,), ())
	def c_py_recurse(m):
	if m:
	_testcapi.pyobject_vectorcall(py_recurse, (1000, m), ())
	depth = sys.getrecursionlimit()
	sys.setrecursionlimit(100_000)
	try:
	recurse(90_000)
	with self.assertRaises(RecursionError):
	recurse(101_000)
	c_recurse(100)
	with self.assertRaises(RecursionError):
	c_recurse(90_000)
	c_py_recurse(90)
	with self.assertRaises(RecursionError):
	c_py_recurse(100_000)
	finally:
	sys.setrecursionlimit(depth)

Looks like the C stack blows up past a depth of 85.

[brettcannon]

@markshannon what's the best way to update this test so it passes with a more limited stack size? Should I update Py_C_RECURSION_LIMIT in Include/cpython/pystate.h to something like 75 when under a debug build for WASI based on the "85" value I found to work?

[brettcannon]

I can also simply skip it much like on x360. I tried setting Py_C_RECURSION_LIMIT to pass various tests that were exhsusting their stacks (e.g., #111803 and #111802), but the limit was getting ridiculously low, to the point that test_pickle was failing on non-deep tests.

[vstinner]

I also saw this issue on Linux (Fedora). But I failed to reproduce it in a reliable way.

cc @hroncok

[vstinner]

When Python is built in debug mode, -Og or event -O0 optimization level is used. It's possible that Python consumes more stack memory and so crash earlier in test_super_deep(): with less deep recursive call.

We already had issues with other tests which rely on how much stack memory is available: https://pythondev.readthedocs.io/unstable_tests.html#unlimited-recursion

Well, I suppose that here the purpose of the test is to prove that Python no longer consumes any stack memory. But maybe this assumption is wrong when Python is built in debug mode?

I propose to reduce the maximum depth is test.support.Py_DEBUG is true. How much? I don't know.

[vstinner]

I can reproduce the issue by building Python with gcc -O0 (disable all compiler optimizations):

./configure --cache-file=../python-config.cache --with-pydebug CFLAGS=-O0
make
./python -m test -v test_call -m test_super_deep 

Output:

vstinner@mona$ ./python -m test -v test_call -m test_super_deep 
== CPython 3.13.0a1+ (heads/main:e5dfcc2b6e1, Nov 15 2023, 19:17:40) [GCC 13.2.1 20231011 (Red Hat 13.2.1-4)]
== Linux-6.5.11-300.fc39.x86_64-x86_64-with-glibc2.38 little-endian
== Python build: debug
== cwd: /home/vstinner/python/main/build/test_python_worker_674715æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1935379936
0:00:00 load avg: 1.79 Run 1 test sequentially
0:00:00 load avg: 1.79 [1/1] test_call
test_super_deep (test.test_call.TestRecursion.test_super_deep) ... Fatal Python error: Segmentation fault

Current thread 0x00007fcd6b199740 (most recent call first):
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  (...)

I tested commit e5dfcc2b6e1c8450e47acbc6e7d3fd3380ef0f03.

[vstinner]

Using gcc -O1, I cannot reproduce the issue: so it seems like Python uses more and less stack memory depending on the compiler optimization level.

[vstinner]

Measure with c_limit=90 py_limit=1,000 recursion_limit=100,000 on Fedora 39 (gcc 13.2.1 on x86-64):

• gcc -O0 + --with-pydebug: c_py_recurse: 13 618.7 bytes/call
• gcc -O1 + --with-pydebug: c_py_recurse: 689.8 bytes/call
• gcc -Og + --with-pydebug: c_py_recurse: 705.6 bytes/call
• gcc -O3 + --with-pydebug: c_py_recurse: 609.6 bytes/call
• gcc -O0: c_py_recurse: 24 690.7 bytes/call
• gcc -Og: c_py_recurse: 577.1 bytes/call
• gcc -O3: c_py_recurse: 464.9 bytes/call

I suppose that the fact that -O0 disable function inlining has a high cost on stack memory consumption.

Script to measure c_py_recurse() stack memory consumption:

import _testcapi
import sys

sys.setrecursionlimit(100_000)
py_limit = 1000
c_limit = 90

lowest = None

def py_recurse(n, m):
    global lowest
    sp = _testcapi.stack_pointer()
    if lowest is None or sp < lowest:
        lowest = sp

    if n >= 1:
        py_recurse(n-1, m)
    else:
        c_py_recurse(m-1)


def c_py_recurse(m):
    global lowest
    sp = _testcapi.stack_pointer()
    if lowest is None or sp < lowest:
        lowest = sp

    if m >= 1:
        _testcapi.pyobject_vectorcall(py_recurse, (py_limit, m), ())


def measure(func, depth):
    global lowest
    lowest = None

    high = _testcapi.stack_pointer()
    func(depth)
    low = lowest
    per_call = (high - low) / depth
    print(f"{func.__name__}: {per_call:.1f} bytes/call "
          f"-- measured with {c_limit=:,} {py_limit=:,} "
          f"recursion_limit={sys.getrecursionlimit():,}")

measure(c_py_recurse, c_limit)

[vstinner]

Latest major change related to recursion:

• GH-107263: Increase C stack limit for most functions, except _PyEval_EvalFrameDefault() #107535
• commit fa45958

This change added test_super_deep() and skips it on s390x architecture.