IndexError on REPL startup in _pyrepl/reader.py with Python 3.13.0b1+ and a custom sys.ps1 set in a PYTHONSTARTUP file

[mgedmin]

Bug report

Bug description:

I compiled Python 3.13 (commit d86b494, the current tip of the 3.13 branch), and installed it into ~/opt/python313. When I run python3.13 with no arguments, I get this:

Python 3.13.0b1+ (heads/3.13:d86b494117, May  9 2024, 15:23:21) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/__main__.py", line 47, in <module>
    interactive_console()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/__main__.py", line 44, in interactive_console
    return run_interactive(mainmodule)
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/simple_interact.py", line 138, in run_multiline_interactive_console
    statement, contains_pasted_code = multiline_input(more_lines, ps1, ps2)
                                      ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/readline.py", line 301, in multiline_input
    return reader.readline(), reader.was_paste_mode_activated
           ~~~~~~~~~~~~~~~^^
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/reader.py", line 646, in readline
    self.prepare()
    ~~~~~~~~~~~~^^
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/historical_reader.py", line 285, in prepare
    super().prepare()
    ~~~~~~~~~~~~~~~^^
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/reader.py", line 521, in prepare
    self.calc_screen()
    ~~~~~~~~~~~~~~~~^^
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/completing_reader.py", line 259, in calc_screen
    screen = super().calc_screen()
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/reader.py", line 299, in calc_screen
    self.cxy = self.pos2xy()
               ~~~~~~~~~~~^^
  File "/home/mg/opt/python313/lib/python3.13/_pyrepl/reader.py", line 474, in pos2xy
    p, l2 = self.screeninfo[y]
            ~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

and it exits with status code 1.

My terminal is currently 126 columns by 45 lines, and I have a custom PYTHONSTARTUP environment file pointing to ~/.python, which you can see in this GH repo. It sets up some conveniences for the old REPL (colorful prompt, rlcompleter-based tab-completion, persistent readline history).

Running PYTHONSTARTUP= python3.13 fixes the IndexError.

By commenting out bits of the file I can reproduce the IndexError with just this:

import sys
sys.ps1 = '\001\033[0;32m\002>>> \001\033[0m\002'
del sys

The \001/\002 characters are hints for readline to inform it that everything between them is a terminal escape sequence that doesn't advance the cursor, without which readline cannot correctly compute the width of the prompt and makes wrong assumptions about line wrapping.

The IndexError goes away if I remove all \001 and \002 characters from sys.ps1.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-118835: pyrepl: Fix prompt length computation for custom prompts containing ANSI escape codes #119942
• [3.13] gh-118835: pyrepl: Fix prompt length computation for custom prompts containing ANSI escape codes (GH-119942) #119990

[mgedmin]

Since I heard that the new REPL code was based on PyPy's, I checked PyPy (7.3.16) and it had no issues with my PYTHONSTARTUP file. I could see that sys.ps1 did include the '\x01'/'\x02' characters.

[The-Compiler]

FWIW I also immediately ran into this when testing 3.13b1. Given that Python 3.13 has its own colored prompt anyways, I guarded it by a:

if sys.version_info != (3, 13, 0, 'beta', 1):

[danielhollas]

Just verified that this is still an issue on the latest main.

[danielhollas]

The problem is somewhere in the _pyrepl/reader.py::Reader.process_prompt method,
which for the following prompt returns length -7. 😱

>>> ps1 = '\001\033[0;32m\002>>> \001\033[0m\002'
>>> Reader.process_prompt(ps1)
('\x1b[0;32m>>> \x1b[0m', -7)

I'll take a closer look tomorrow.