string length overflows to negative when executing

[mqyhlkahu]

Bug report

Checklist

• I am confident this is a bug in CPython,
  not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

A clear and concise description of the bug

The following code causes an error to be raised in all tested versions of Python.

exec(f"if True:\n {' ' * 2**31}print('hello world')")

Which error is raised varies by version, and as such the specific error message is listed with its corresponding version in the below section.

Some (hopefully) helpful information:

• I have confirmed that the creation of the string is not causing a problem (by creating it separately, storing it in a variable, and then exec'ing it), the problem occurs when attempting to exec it.
• I have confirmed that the issue is does not occur solely because of the large string. (The same issue does not occur when I use a value of, for example, 2**33).

NOTE:

I believe that this error will also occur when running real file rather than building a string and using exec, but I have not yet confirmed that.

Your environment

I tested this in three environments, with as many versions of Python as I could conveniently use.

Arch GNU/Linux

• Operating system and architecture:
  • Operating system: Arch GNU/Linux
  • CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz 2.80 GHz
• CPython versions:
  • Python 3.10.10 (main, Mar 5 2023, 22:56:53) [GCC 12.2.1 20230201]
    • IndentationError: unindent does not match any outer indentation level
  • Python 3.11.3 (main, Jun 5 2023, 09:32:32) [GCC 13.1.1 20230429]
    • SystemError: Negative size passed to PyUnicode_New

Microsoft Windows 10

• Operating system and architecture:
  • Operating system: Microsoft Windows 10 (version 22H2 build 19045.3208) inside a VirtualBox virtual machine
  • CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz 2.80 GHz
• CPython versions:
  • Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 16:30:00) [MSC v.1900 64 bit (AMD64)]
    • IndentationError: None
  • Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
    • IndentationError: None
  • Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]
    • IndentationError: unindent does not match any outer indentation level
  • Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
    • IndentationError: unindent does not match any outer indentation level
  • Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]
    • SystemError: Negative size passed to PyUnicode_New

Ubuntu GNU/Linux

• Operating system and architecture:
  • Operating system: Ubuntu 22.04 GNU/Linux inside a VirtualBox virtual machine
  • CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz 2.80 GHz
• CPython versions:
  • Python 3.7.17 (default, Jun 6 2023, 20:10:09) [GCC 11.3.0]
    • IndentationError: None
  • Python 3.8.17 (default, Jun 6 2023, 20:10:50) [GCC 11.3.0]
    • IndentationError: None
  • Python 3.9.17 (main, Jun 6 2023, 20:11:21) [GCC 11.3.0]
    • IndentationError: unindent does not match any outer indentation level
  • Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
    • IndentationError: unindent does not match any outer indentation level
  • Python 3.11.4 (main, Jun 7 2023, 12:45:48) [GCC 11.3.0]
    • SystemError: Negative size passed to PyUnicode_New

Should any further information be required about the environments that I used, please ask.

What I believe causes the problem

I suspect very strongly that the error occurs as a result of an integer overflow. (A value larger than 2**31 - 1 will overflow and become negative when using 2's complement with 32 bit signed integers).

Possible solution

I think that this can be fixed by using a 64 bit integer to hold the length of the string while 64 bit platforms. This, while still leaving the error theoretically possible, would in practice avoid the error entirely because one would encounter a MemoryError first.

Linked PRs

• gh-107450: Raise OverflowError when parser column offset overflows #110754
• [3.12] gh-107450: Raise OverflowError when parser column offset overflows (GH-110754) #110762
• [3.11] gh-107450: Raise OverflowError when parser column offset overflows (GH-110754) #110763
• gh-107450: Fix parser column offset overflow test on Windows #110768
• GH-107450: fix unexpected OverflowError in testMemoryErrorBigSource #110801
• [3.12] gh-107450: Fix parser column offset overflow test on Windows (GH-110768) #110808
• [3.11] gh-107450: Fix parser column offset overflow test on Windows (GH-110768) #110809
• gh-107450: Fix testMemoryErrorBigSource using the bigmemtest decorator #110810
• gh-107450: Check for overflow in the tokenizer and fix overflow test #110832
• gh-107450: Fix testMemoryErrorBigSource in test_exceptions #110871
• [3.12] gh-107450: Check for overflow in the tokenizer and fix overflow test (GH-110832) #110931
• [3.11] gh-107450: Check for overflow in the tokenizer and fix overflow test (GH-110832) #110939
• gh-107450: Remove unnecessary overflow check in parser error handler #110940

[terryjreedy]

There have been multiple reports that extreme artificial code produces compile problems or runtime problems, either exceptions or crashes. The answer for some is "don't do that; all compilers have limits".

'exec' compiles and runs. IndentationErrors are compile errors. Installed 3.12.0b4 Win 10, running compile on IDLE gives the same SystemError. Fresh 3.13.0a0 debug REPL crashes after perhaps a minute with C Runtime debug error: abort() has been called.

[serhiy-storchaka]

I agree that "don't do that" is the answer, but SystemError should never be triggered by the user Python code. It signals about improper use of the C API or some issues in the core. Although the former IndentationError was also not very useful. An error should explicitly report about the compiler limitation, not rely on the undefined behavior of integer overflow in C.

On main in debug build it crashes after long delay.

python: Parser/pegen_errors.c:237: _PyPegen_raise_error: Assertion `_Py_STATIC_CAST(intptr_t, _Py_STATIC_CAST(int, (p->tok->cur - start))) == (p->tok->cur - start)' failed.
Aborted (core dumped)

[mqyhlkahu]

Thank you both for your response!

There have been multiple reports that extreme artificial code produces compile problems or runtime problems, either exceptions or crashes. The answer for some is "don't do that; all compilers have limits".

Yes, I am aware that compilers have limits, and I don't think that's a bad thing! It would take a lot more work, and quite probably be a lot slower, were these limitations to be removed.

However, I am of the opinion that:

1. SystemErrors shouldn't be raised because of user code (except for raise SystemError and the like, of course).
2. The previous IndentationErrors are also bad, because they don't really explain the problem (and, in the case of 3.9 / 3.10, actively mislead the user).

While I would certainly prefer this to be fixed, I understand that it probably won't be. However, I feel that this should at least provide a useful error message (and not provide a SystemError), perhaps something along the lines of the exception that results from

raise BaseException("compiler error") from OverflowError("string length overflowed")

It would hopefully utilise BaseException so that the error is less likely to be caught unintentionally by someone. (Ideally, there would be an exception-type derived from BaseException called CompilerError or something like that, but introducing a new exception-type is FAR beyond the scope of this issue).

[serhiy-storchaka]

The test should be decorated with @bigmemtest.