pdb cannot find source code for frozen stdlib modules

[SnoopJ]

Bug report

Context: this issue came out of a discussion in #python on the Libera.chat IRC network, where a user wanted to peek into importlib._bootstrap with pdb while chasing a bug.

pdb is capable of stepping into function calls into frozen modules, but the list command cannot locate the source code necessary to display the source being stepped through.

# repro.py
import importlib._bootstrap

# some function call that we want to step into with pdb
importlib._bootstrap._resolve_name("os", ".", 1)

$ python3 -m pdb repro.py
> /home/snoopjedi/repro.py(2)<module>()
-> import importlib._bootstrap
(Pdb) n
> /home/snoopjedi/repro.py(5)<module>()
-> importlib._bootstrap._resolve_name("os", ".", 1)
(Pdb) s
--Call--
> <frozen importlib._bootstrap>(883)_resolve_name()
(Pdb) l
[EOF]

Note that executing source importlib._bootstrap from the frame that calls into this module does successfully locate the source, but this isn't very useful to a user of pdb.

I believe that bringing the frame's co_filename into agreement with the module's __file__ would fix this issue without changes to pdb (see #89815), but thought it would be good to track the issue with pdb in a separate ticket since that fix is more nuanced and I think I have an interim patch for the debugger.

Your environment

• CPython versions tested on: 3.9.4, 3.11.0b3
• Operating system and architecture: Ubuntu 20.04, x86_64

Linked PRs

• gh-93696: Fixed the breakpoint display error for frozen modules #132862

[jacobtylerwalls]

Likely duplicate of #60115

[SnoopJ]

Likely duplicate of #60115

Definitely related inasmuch as importlib._bootstrap is involved, @exarkun is the user I mentioned in my report. Note that the issue does happen with other frozen modules, though, so the scope of the problem is expanding in 3.11 because of #89183. Here's an example showing the same problem with os.path

# repro2.py
import os.path

os.path.basename("/foo/bar/baz.py")  # implemented in Python

With Python 3.10:

$ python3.10 -m pdb repro2.py 
> /home/snoopjedi/repos/cpython/repro2.py(2)<module>()
-> import os.path
(Pdb) n
> /home/snoopjedi/repos/cpython/repro2.py(4)<module>()
-> os.path.basename("/foo/bar/baz.py")  # implemented in Python
(Pdb) s
--Call--
> /home/snoopjedi/.pyenv/versions/3.10.0/lib/python3.10/posixpath.py(140)basename()
-> def basename(p):
(Pdb) l
135         return p[:0], p
136  
137  
138     # Return the tail (basename) part of a path, same as split(path)[1].
139  
140  -> def basename(p):
141         """Returns the final component of a pathname"""
142         p = os.fspath(p)
143         sep = _get_sep(p)
144         i = p.rfind(sep) + 1
145         return p[i:]

With Python 3.11:

$ python3.11 -m pdb repro2.py 
> /home/snoopjedi/repos/cpython/repro2.py(2)<module>()
-> import os.path
(Pdb) n
> /home/snoopjedi/repos/cpython/repro2.py(4)<module>()
-> os.path.basename("/foo/bar/baz.py")  # implemented in Python
(Pdb) s
--Call--
> <frozen posixpath>(140)basename()
(Pdb) l
[EOF]
(Pdb)

[SnoopJ]

Pinging this issue per the devguide in the hopes of getting review on the PR, which has been open for a little while. I understand if things are too busy for review because of the release issues with 3.11, though!

[ericsnowcurrently]

This should mostly be resolved in 3.10. See issue gh-65935 (and PR gh-28656). We now set __file__ for frozen stdlib modules. Could you give it a try?