bpo-42988: Fix security issue in the pydoc server

[serhiy-storchaka]

Route /getfile?key= no longer allows to get content of arbitrary file.
It serves now only source files of Python modules available to pydoc.

https://bugs.python.org/issue42988

[vstinner]

Maybe this code can be factorized with ModuleScanner.run()?

[serhiy-storchaka]

Unfortunately it cannot. ModuleScanner.run() contains too much specific to its primary function: it only calls callback if key is not None and is found in module name or description.

But it is a temporary solution. For 3.10 we can change API (pass not path, but a module name, so the loop will be unnecessary).

[Fidget-Spinner]

I ran some tests on large codebases and I'm not sure why, but this doesn't seem to work when I launch it in conda environment with a pip installed editable mode local package. It seems that pkgutil.walk_packages() doesn't find the pip dependencies of editable package.

Granted this is a weird scenario, but I would think people want to see the docs of their own package they're developing locally.

Currently some example code that triggers it:
setup.py:

import setuptools
setuptools.setup(
    name="foo",
    version="0.0.0",
    author="a",
    author_email="b",
    description="Sample",
    long_description="Sample text",
    long_description_content_type="text/markdown",
    url="https://example.com",
    packages=[""],
    include_package_data=True,
    install_requires=["regex"],
    classifiers=[],
    python_requires='>=3.7',
)

__init__.py:

import regex

Then pydoc -b for this fails:

import foo

I'll report back if it affects venv or other stuff, maybe it's just my computer and it can't be reproduced anywhere else.

[Fidget-Spinner]

pkgutil.walk_packages() can seemingly fail if a module has no __version__. Eg. on my computer I get AttributeError for some weird packages:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "......\lib\pkgutil.py", line 92, in walk_packages
    __import__(info.name)
  File "......\lib\site-packages\brotli_asgi\__init__.py", line 12, in <module>
    brotli = import_brotli()
  File "......\lib\site-packages\brotli_asgi\_import_resolver.py", line 22, in import_brotli
    spec.loader.exec_module(brotli)
  File "......\lib\site-packages\brotli.py", line 12, in <module>
    __version__ = _brotli.__version__
AttributeError: module 'brotli._brotli' has no attribute '__version__'

Because:

>>> import brotli
>>> brotli._brotli
<module 'brotli._brotli' from '......\\lib\\site-packages\\brotli\\_brotli.cp38-win_amd64.pyd'>
>>> brotli._brotli.__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'brotli._brotli' has no attribute '__version__'

[serhiy-storchaka]

Thank you. Fixed.

[kam193]

Hi, I've randomly opened this PR. I'd like to ask you to check this error message, it looks like unfinished.

[vstinner]

Walking into all packages sounds overkill to me. Moreover, it doesn't prevent to read sensitiva data from Python modules source code.

I proposed a simpler approach in PR #25015: just remove the feature.

[vstinner]

I merged PR #25015 fix instead.