Zero-width characters cause unbounded memory use in PyREPL

[trag1c]

Bug report

Bug description:

Trying to paste zero-width characters (U+200B, U+200C, U+200D, U+FEFF) into the new REPL causes it to get stuck and keep allocating memory. Python 3.12 handles them fine.

This seems to happen during screen calculation, so the amount of characters needed to reproduce will depend on the number of cells in a terminal emulator window (I got it working with as few as 3 with a big enough zoom, but 100 should be enough).

A table with all of them x100 to copy for convenience:

ZWSP	ZWNJ	ZWJ	ZWNBSP
​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​	‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌	‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍	

To reproduce

1. Open the new REPL
2. Paste in some zero-width characters
3. The REPL should get stuck and keep allocating memory
4. Can be stopped with double ^C or a single ^\

The culprit seems to be this loop in Lib/_pyrepl/reader.py:Reader.calc_screen and its screen/screen_info appends:

cpython/Lib/_pyrepl/reader.py

Lines 388 to 411 in f4e5643

	while l:
	prelen = lp if i == 0 else 0
	index_to_wrap_before = 0
	column = 0
	for character_width in l2:
	if column + character_width >= self.console.width - prelen:
	break
	index_to_wrap_before += 1
	column += character_width
	pre = prompt if i == 0 else ""
	if len(l) > index_to_wrap_before:
	offset += index_to_wrap_before
	post = "\\"
	after = [1]
	else:
	offset += index_to_wrap_before + 1 # Takes the newline
	post = ""
	after = []
	last_refresh_line_end_offsets.append(offset)
	screen.append(pre + l[:index_to_wrap_before] + post)
	screeninfo.append((prelen, l2[:index_to_wrap_before] + after))
	l = l[index_to_wrap_before:]
	l2 = l2[index_to_wrap_before:]
	i += 1

An additional demo

Screen.Recording.2024-11-11.at.13.52.32.mov

CPython versions tested on:

3.12, 3.13, CPython main branch

Operating systems tested on:

Linux, macOS

[bswck]

Excellent find. Needs a backport to 3.13. Gonna fix soon.

[bswck]

@ambv That's the bug we discussed at Python Łódź, could you please assign me?
I'll try to get it fixed on Friday.

[bswck]

Automatic reproduction for Unix:

# gh-126685.py

import os
import signal
from _pyrepl.unix_console import UnixConsole
from typing import Any
from time import sleep
from threading import Thread

mocked_term_width = 8

UnixConsole.getheightwidth = lambda self: (
    mocked_term_width,
    os.get_terminal_size(self.input_fd)[1],
)


def new_init(self: UnixConsole, /, *args: Any, **kwargs: Any) -> None:
    UnixConsole_init(self, *args, **kwargs)
    self.input_buffer = b"# " + b"\xe2\x80\x8b" * (mocked_term_width + 1) + b"\nexit\n"


UnixConsole_init = UnixConsole.__init__
UnixConsole.__init__ = new_init


Thread(
    target=lambda: sleep(30) or os.kill(os.getpid(), signal.SIGKILL),
    daemon=True,
).start()

run with PYTHONSTARTUP=gh-126685.py python -q.

exit not written as a prompt means the bug was reproduced, zero status means converse.

[ambv]

Fixed by #131547 and #131557.