test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler?

[serhiy-storchaka]

Bug report

Every test class in test_httpservers starts a thread to run a server. CGIHTTPServerTestCase tests trigger htt.server.CGIHTTPRequestHandler.run_cgi() which executes a CGI script in a separate process. Using os.fork() if possible. But os.fork() now emits a DeprecationWarning if used in multi-threaded environment, and in future it will be an error.

$ ./python -Wa -m test -v test_httpservers -m CGIHTTPServerTestCase
...
test_accept (test.test_httpservers.CGIHTTPServerTestCase.test_accept) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
ok
test_authorization (test.test_httpservers.CGIHTTPServerTestCase.test_authorization) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
ok
test_cgi_path_in_sub_directories (test.test_httpservers.CGIHTTPServerTestCase.test_cgi_path_in_sub_directories) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
ok
...

We should do something with this. First, the tests are polluted with presumably false alerts. Second, these alerts may be not false. Third, these tests will be broken in future, when the warning is turned into error.

Also, what actions are expected from users of http.server when they see a warning? How they can solve the problem on their side? If http.server is deprecated, it should be documented. If it remains, it should not produce scary warnings.

cc @gpshead

Linked PRs

• gh-109096: Deprecate http.server.CGIHTTPRequestHandler #109387
• [3.12] gh-109096: Silence test_httpservers fork + threads DeprecationWarning on CGI support #109471

[gpshead]

At a minimum we should at least silence this particular warning in this particular CGIHTTPServerTestCase class. If this part of the test suite starts hanging on particular platform configurations due to the technically not allowed by posix issue, we can add explicit skips for those. To fix the underlying issue and not silence it:

The real bug exposed is that http.server.CGIHTTPRequestHandler has never been safe to use on a multithreaded server. This code - https://github.com/python/cpython/blob/main/Lib/http/server.py#L1172-L1192 - where the warning comes from needs to be rewritten to use subprocess.

But given that http.server is not considered a good idea to use in serious applications (aka "production") per the warning at the top of its docs, nobody has been paying attention to that kind of maintenance on it. It does not see much use. https://docs.python.org/3/library/http.server.html

[serhiy-storchaka]

We should not simply silence deprecation warnings. We should make os.fork() only be used if it is safe (only in single-threaded program) or allow the user to control this by an option. We should rewrite tests and test the fork-using code without threads, and subprocess-using code with threads. We should document the limitations of using CGI in http.server.

[gpshead]

I wonder if we need CGIHTTPRequestHandler at all? It isn't the '90s any more, CGI is... extremely dated and often problematic. Deprecating it could be another option.

[gpshead]

the cgi and cgitb modules were removed in 3.13 for https://peps.python.org/pep-0594/#deprecated-modules.

Searching internally (which covers a large PyPI/github third_party corpus as well) I only found a single use of CGIHTTPRequestHandler in actual code in an internal demo test server. It was a misunderstanding, they didn't need to use that class and should've used SimpleHTTPRequestHandler as they implemented their own do_POST method.

[gpshead]

If anyone can find actual uses of CGIHTTPRequestHandler in real world code, it'd be good to see what still uses it and why.

[gpshead]

The deprecation is in for 3.13. I'll work up a small 3.12 specific test-only change to silence the warning on the CGIHTTPRequestHandler test there.