Null characters in strings cause a C SystemError

[apccurtiss]

Crash report

Putting a null byte into a Python string causes a SystemError in Python 3.10, due to a call to strlen in the string parsing library. In Python 3.9, the following example runs without errors:

# -*- coding: latin-1 -*-
"""
<NULL>
"""

In Python 3.10, it raises SystemError: ../Parser/string_parser.c:219: bad argument to internal function.

Internally, the new string_parser library introduced in v3.10.0a1 uses a call to strlen to determine the string size, which is getting thrown off by the null byte. This call is actually unnecessary, as the length has already been calculated by the calling parser and can be retrieved with PyBytes_AsStringAndSize.

Error messages

For single line strings, the error is SystemError: Negative size passed to PyUnicode_New

For multiline strings, the error is SystemError: ../Parser/string_parser.c:219: bad argument to internal function

Linked PRs

• gh-97556: Raise null bytes syntax error upon null in multiline string #104136

[vstinner]

This issue seems like a duplicate of the issue #96670.

[apccurtiss]

Ah, you're totally right. I found that one earlier, but misunderstood the core bug and re-discovered it. Thanks!

[apccurtiss]

Actually, I just re-read that issue, and I was a bit over-eager with closing this. I feel like this is a very specific fix which doesn't actually solve their use case: This particular issue only shows up in 3.10, and causes an internal parse error. That linked issue is present in every 3.x version I've tested, including after this fix. I believe a patch elsewhere in the parser is needed for that.

[gvanrossum]

Can you submit a fix?

[apccurtiss]

Submitted a fix in this pull request

[bskinn]

I cannot repro OP's SystemError on this. I do observe Python 3.12's fix for null characters in single-quoted strings (SyntaxError: source code cannot contain null bytes per #97594, when run with 3.12.0a7), so going forward I'm focusing on triple-quoted strings.

For Python 3.8.16, 3.9.16, and 3.10.0a1 (WSL Debian local builds), crasher.py with the following contents parses successfully (without error, I did not check correctness) when executed from the commandline ($ python3.<#> crasher.py):

<BLANKLINE>
'''
<NUL>
'''

For Python 3.10.0b1 (WSL Debian local), 3.10.0 (WSL Debian local), 3.10.11 (WSL Debian local and Win11), 3.11.3 (WSL Debian local and Win11), and (importantly) 3.12.0a7 (WSL Debian local), parsing fails with SyntaxError: unterminated triple-quoted string literal.

Interestingly, the attributed line number of the exception location changes in 3.12.0a7. It's reported as line 3 in 3.12.0a7, and as line 2 for all other versions tested.

Regardless, the triple-quoted string case is not yet fixed in the fashion indicated by #96670 (comment), where strings containing null characters are explicitly rejected with an exception and message like SyntaxError: source code cannot contain null bytes.