Is it OK to implement stack protection?

[markshannon]

C stack overflow has lately been a problem for building and testing CPython, and it is a concern that it may impact users as well.
The root cause is that compilers, especially Clang use a lot more C stack for debug builds, meaning that at the full default recursion limit of 1000, C stack overflows are likely.

I would like to fix CPython to raise an exception (for major platforms at least) instead of faulting. We already do this partially for Windows.

PEP 651 proposed doing this, but was rejected. I don't want to appear to be implementing PEP 651 by stealth, so I wanted the SC's approval first.

PEP 651 included the following:

StackOverflow exception

A StackOverflow exception will be raised whenever the interpreter or builtin module code determines that the C stack is at or nearing a limit of safety. StackOverflow is a sub-class of RecursionError, so any code that handles RecursionError will handle StackOverflow.

I plan to implement the above check, but without adding a new Exception type, raising a RecursionError as happens when the recursion limit is reached.

[brettcannon]

I've added it as the 19th item on our agenda. 😅

[markshannon]

Thanks.

[Yhg1s]

The SC is fine with adding stack protection and raising an exception instead of crashing, as long as the mechanisms are either portable or conditional on platform support. However, the exception should be MemoryError instead of RecursionError; overflowing the stack isn't necessarily caused by recursion, and we don't want to confuse the two situations. @gpshead would like to review the PRs that implement this, as he has some experience in this area.

[dimpase]

I believe that python/cpython#96510 has not completely implemented the resolution by SC, e.g. no MemoryError is thrown in case of C stack overflow, but RecursionError.

See also python/cpython#112282 and python/cpython#112215 pointing at related issues that this has caused.

[markshannon]

python/cpython#91079 (comment) explains why the exception must be a subclass of RecursionError, not MemoryError.