bpo-42988: Remove the pydoc getfile feature

[vstinner]

CVE-2021-3426: Remove the "getfile" feature of the pydoc module which
could be abused to read arbitrary files on the disk (directory
traversal vulnerability). Moreover, even the Python modules source
code can contain sensitive data like passwords. Vulnerability
reported by David Schwörer.

https://bugs.python.org/issue42988

[vstinner]

I kept links to the source file using file:// scheme. For example, http://localhost:8080/argparse.html creates /home/vstinner/python/master/Lib/argparse.py link to the URL: file:///home/vstinner/python/master/Lib/argparse.py

file:///home/vstinner/python/master/Lib/argparse.py

This doesn't work if the pydoc server is accessed from a different machine, but the common usage (IMO) is to only run pydoc locally since pydoc only listen on the local link (localhost) by default.

[vstinner]

I kept links to the source file using file:// scheme

These links don't work when the pydoc server is accessed from a different machine. An alternative is to simply remove these links and let the user open the file manually, since the path is given in the HTML page (just copy/paste the path).

[dmerejkowsky]

My two cents: this looks like an acceptable compromise to me.

A quick survey among other Pythonistas I know show that the pydoc feature is seldomly used, the HTTP server even less so, and the getfile feature even less.

Removing a vulnerable feature that almost nobody uses is much less work than trying to fix the bug (the other PRs trying to fix it already add quite a bit of complexity - and who knows what else the attackers will find ?)

I would suggest to consider fixing the bugs instead of removing the feature only if enough people complain.

[vstinner]

@Fidget-Spinner @serhiy-storchaka: Would you mind to review this PR? Are you ok with this compromise? https://bugs.python.org/issue42988#msg389452

[Fidget-Spinner]

Would you mind to review this PR?

From a technical point of view, this PR looks good. Thanks for taking the time to do this Victor!

Are you ok with this compromise? https://bugs.python.org/issue42988#msg389452

I'm waiting for some consensus on the bug tracker by the core devs. Personally I don't really have a strong opinion, so a +0 ;) - I wouldn't oppose if this were accepted.

[willingc]

Thanks @vstinner. This looks like a reasonable and safe approach.

[JulienPalard]

I had 4 other ways to fix it in https://bugs.python.org/issue42988, didn't though about removing it... but I'm happy with removing it! It keeps things simple.

The other ways adds too much complexity that do not balance well with the usage of this feature (I, too, suspect nobody uses this).

[vstinner]

@serhiy-storchaka: I would prefer to get your opinion on this PR, but I don't want to wait too long since https://bugs.python.org/issue42988 vulnerability was already reported 3 months ago and got a CVE number. The PR already got 3 approvals. I plan to merge this PR next week (not sure when exactly). Tell me if you want me to wait for your review or not.

[miss-islington]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.6, 3.7, 3.8, 3.9.
🐍🍒⛏🤖

[bedevere-bot]

GH-25064 is a backport of this pull request to the 3.9 branch.

[bedevere-bot]

GH-25065 is a backport of this pull request to the 3.8 branch.

[vstinner]

Thanks for the reviews! I love fixing security vulnerabilities by removing code!

[bedevere-bot]

GH-25066 is a backport of this pull request to the 3.7 branch.

[bedevere-bot]

GH-25067 is a backport of this pull request to the 3.6 branch.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL7 LTO + PGO 3.x has failed when building commit 9b99947.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/244/builds/931) and take a look at the build logs.
4. Check if the failure is related to this commit (9b99947) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/244/builds/931

Failed tests:

• test_tools

Failed subtests:

• test_reindent_file_with_bad_encoding - test.test_tools.test_reindent.ReindentTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

• test_concurrent_futures: 2 min 34 sec
• test_multiprocessing_spawn: 1 min 26 sec
• test_tokenize: 1 min 21 sec
• test_multiprocessing_forkserver: 1 min 10 sec
• test_unparse: 1 min 9 sec
• test_nntplib: 1 min 6 sec
• test_multiprocessing_fork: 1 min
• test_asyncio: 59.1 sec
• test_lib2to3: 58.1 sec
• test_signal: 48.3 sec

1 test failed:
test_tools

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_tools

Total duration: 6 min 46 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/test_tools/test_reindent.py", line 29, in test_reindent_file_with_bad_encoding
    rc, out, err = assert_python_ok(self.script, '-r', bad_coding_path)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/python', '-X', 'faulthandler', '-I', '/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Tools/scripts/reindent.py', '-r', '/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/bad_coding.py']