The new repl sometimes does not show the error location when `SyntaxError` is raised

[tomasr8]

Bug report

Bug description:

When I run def f[Foo, Foo](): ... from a file, I get a nice error message including the location of the error:

  File ".../test.py", line 1
    def f[Foo, Foo](): ...
               ^^^
SyntaxError: duplicate type parameter 'Foo'

However, when I run this in the new repl, the error location is not shown:

>>> def f[Foo, Foo](): ...
  File "<python-input-0>", line 1
SyntaxError: duplicate type parameter 'Foo'

The same thing happens with this error as well:

>>> def f(x, x): ...
  File "<python-input-0>", line 1
SyntaxError: duplicate argument 'x' in function definition

The missing location seems to be the case for these two errors only as other errors do show it:

>>> def f(42): ...
  File "<unknown>", line 1
    def f(42): ...
          ^^
SyntaxError: invalid syntax

What's interesting is that for the first two errors the file shows up as <python-input-0> while for the last one it's <unknown>. This could be related to the last error coming from the parser while the previous errors come from symtable:

cpython/Python/symtable.c

Lines 1421 to 1435 in 6522f0e

	if ((o = PyDict_GetItemWithError(dict, mangled))) {
	val = PyLong_AS_LONG(o);
	if ((flag & DEF_PARAM) && (val & DEF_PARAM)) {
	/* Is it better to use 'mangled' or 'name' here? */
	PyErr_Format(PyExc_SyntaxError, DUPLICATE_ARGUMENT, name);
	SET_ERROR_LOCATION(st->st_filename, loc);
	goto error;
	}
	if ((flag & DEF_TYPE_PARAM) && (val & DEF_TYPE_PARAM)) {
	PyErr_Format(PyExc_SyntaxError, DUPLICATE_TYPE_PARAM, name);
	SET_ERROR_LOCATION(st->st_filename, loc);
	goto error;
	}
	val |= flag;
	}

Though it looks like that the error location is being set with SET_ERROR_LOCATION..

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-121804: always show error location for SyntaxError's in new repl #121886
• [3.13] gh-121804: Always show error location for SyntaxError's in new repl (GH-121886) #123148
• gh-121804: always show error location for SyntaxError's in basic repl #123202
• [3.12] gh-121804: Backport idlelib.pyshell change (GH-121886) #123366
• [3.13] gh-121804: always show error location for SyntaxError's in basic repl (GH-123202) #123631

[skirpichev]

This issue is valid for basic repl as well:

$ PYTHON_BASIC_REPL=1 ./python
Python 3.14.0a0 (heads/main:bf74db731b, Jul 15 2024, 11:06:00) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def f[Foo, Foo](): ...
...
  File "<stdin>-0", line 1
SyntaxError: duplicate type parameter 'Foo'
>>> def f(x, x): ...
... 
  File "<stdin>-1", line 1
SyntaxError: duplicate argument 'x' in function definition

[sobolevn]

These errors happen in different places (parser vs symtable):

// symtable.c:1425
PyErr_Format(PyExc_SyntaxError, DUPLICATE_ARGUMENT, name);
SET_ERROR_LOCATION(st->st_filename, loc);

So, I guess it is not related to the new repl.

[skirpichev]

Exceptions metadata differ when compile() has valid filename argument wrt fake filename like <input>. It seems, the only way to workaround this is using a temporary file to dump code:

>>> def f(x, x): ...
  File "/tmp/tmpuygzaco0", line 1
    def f(x, x): ...
             ^
SyntaxError: duplicate argument 'x' in function definition

POC diff

diff --git a/Lib/_pyrepl/console.py b/Lib/_pyrepl/console.py
index a8d3f52034..0df16c2b3d 100644
--- a/Lib/_pyrepl/console.py
+++ b/Lib/_pyrepl/console.py
@@ -168,6 +168,12 @@ def showtraceback(self):
         super().showtraceback(colorize=self.can_colorize)
 
     def runsource(self, source, filename="<input>", symbol="single"):
+        import tempfile
+        import os
+        file = tempfile.NamedTemporaryFile('w+', delete=False)
+        file.write(source)
+        file.close()
+        filename = file.name
         try:
             tree = ast.parse(source)
         except (SyntaxError, OverflowError, ValueError):
@@ -189,11 +195,15 @@ def runsource(self, source, filename="<input>", symbol="single"):
                         f" top-level 'await' and run background asyncio tasks."
                     )
                 self.showsyntaxerror(filename)
+                os.unlink(filename)
                 return False
             except (OverflowError, ValueError):
                 self.showsyntaxerror(filename)
+                os.unlink(filename)
                 return False
 
+            os.unlink(filename)
+
             if code is None:
                 return True
 

[sobolevn]

There are multiple systems / setups where temporary files could not be created:

• readonly mounts
• permissions
• etc

Plus, I am not sure that users expect to see File "/tmp/tmpuygzaco0", line 1, I think that it should be a bit more complicated:

• We can try to create and open this temp file, if not - fallback to default behaviour
• If it is opened, then still show an old <input> tag as a visible filename

There's also a question about how to append / rewrite / recreate this file on new statements.

Storing state is hard :(

[tomasr8]

imo fixing the core issue with the metadata is probably better than using a temp file for the reasons @sobolevn mentioned, but I don't know enough about the code to tell how hard that would be..

[skirpichev]

There are multiple systems / setups where temporary files could not be created

Yes, this is not easy:)

Plus, I am not sure that users expect to see File "/tmp/tmpuygzaco0", line 1

I think this can be fixed.

On another hand, we have almost all required metadata for exception in it's attributes:

>>> try:
...     compile("def f(x,x): ...\n", "spam.py", "single")
... except Exception as e:
...     print(e.lineno, e.offset)
...     
1 9

The text attribute is empty.

So, we can provide source as an optional kwarg for showsyntaxerror() method. Given this, we can "fix" empty text attribute of the exception. With this patch (see details below):

>>> def f(x, x): ...
  File "<python-input-0>", line 1
    def f(x, x): ...
             ^
SyntaxError: duplicate argument 'x' in function definition
>>> def f[Foo, Foo](): ...
  File "<python-input-1>", line 1
    def f[Foo, Foo](): ...
               ^^^
SyntaxError: duplicate type parameter 'Foo'

POC patch

diff --git a/Lib/_pyrepl/console.py b/Lib/_pyrepl/console.py
index a8d3f52034..1eda92d49c 100644
--- a/Lib/_pyrepl/console.py
+++ b/Lib/_pyrepl/console.py
@@ -161,8 +161,8 @@ def __init__(
         super().__init__(locals=locals, filename=filename, local_exit=local_exit)  # type: ignore[call-arg]
         self.can_colorize = _colorize.can_colorize()
 
-    def showsyntaxerror(self, filename=None):
-        super().showsyntaxerror(colorize=self.can_colorize)
+    def showsyntaxerror(self, filename=None, **kwargs):
+        super().showsyntaxerror(colorize=self.can_colorize, **kwargs)
 
     def showtraceback(self):
         super().showtraceback(colorize=self.can_colorize)
@@ -188,7 +188,7 @@ def runsource(self, source, filename="<input>", symbol="single"):
                         f"Try the asyncio REPL ({python} -m asyncio) to use"
                         f" top-level 'await' and run background asyncio tasks."
                     )
-                self.showsyntaxerror(filename)
+                self.showsyntaxerror(filename, source=source)
                 return False
             except (OverflowError, ValueError):
                 self.showsyntaxerror(filename)
diff --git a/Lib/code.py b/Lib/code.py
index a55fced070..78bcdcdca5 100644
--- a/Lib/code.py
+++ b/Lib/code.py
@@ -64,7 +64,7 @@ def runsource(self, source, filename="<input>", symbol="single"):
             code = self.compile(source, filename, symbol)
         except (OverflowError, SyntaxError, ValueError):
             # Case 1
-            self.showsyntaxerror(filename)
+            self.showsyntaxerror(filename, source=source)
             return False
 
         if code is None:
@@ -107,6 +107,7 @@ def showsyntaxerror(self, filename=None, **kwargs):
 
         """
         colorize = kwargs.pop('colorize', False)
+        source = kwargs.pop('source', "")
         type, value, tb = sys.exc_info()
         sys.last_exc = value
         sys.last_type = type
@@ -124,6 +125,8 @@ def showsyntaxerror(self, filename=None, **kwargs):
                 value = SyntaxError(msg, (filename, lineno, offset, line))
                 sys.last_exc = sys.last_value = value
         if sys.excepthook is sys.__excepthook__:
+            if source:
+                value.text = source.splitlines()[value.lineno-1]
             lines = traceback.format_exception_only(type, value, colorize=colorize)
             self.write(''.join(lines))
         else:

If this is not too hackish way to deal with the issue - I'll make a pr.

Edit:

Similar fix is possible for basic repl.

POC patch for basic repl

diff --git a/Python/pythonrun.c b/Python/pythonrun.c
index ce7f194e92..43e12acb0e 100644
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -283,6 +283,19 @@ PyRun_InteractiveOneObjectEx(FILE *fp, PyObject *filename,
     _PyArena_Free(arena);
     Py_DECREF(main_module);
     if (res == NULL) {
+        PyThreadState *tstate = _PyThreadState_GET();
+        PyObject *exc = tstate->current_exception;
+        if ((PyObject *)Py_TYPE(exc) == PyExc_SyntaxError) {
+            /* fix "text" attribute */
+            PyObject *xs = PyUnicode_Splitlines(interactive_src, 1);
+            PyObject *ln = PyObject_GetAttr(exc, &_Py_ID(lineno));
+            PyObject *line = PyList_GetItem(xs, PyLong_AsLong(ln) - 1);
+            Py_DECREF(ln);
+            Py_INCREF(line);
+            Py_DECREF(xs);
+            PyObject_SetAttr(exc, &_Py_ID(text), line);
+            Py_DECREF(line);
+        }
         return -1;
     }
     Py_DECREF(res);

[skirpichev]

pr is ready for review: #121886

[skirpichev]

cc @pablogsal, maybe you could take look on pr?

[pablogsal]

Will do today