[tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634)

[vstinner]

First failure: https://buildbot.python.org/#/builders/484/builds/6224

Example of failure:

test_stls_context (test.test_poplib.TestPOP3Class.test_stls_context) ...

Warning -- Uncaught thread exception: SSLError
Exception in thread Thread-16:
Traceback (most recent call last):
  (...)
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/test_poplib.py", line 203, in handle_read
    asynchat.async_chat.handle_read(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/support/asynchat.py", line 128, in handle_read
    self.handle_error()
    ~~~~~~~~~~~~~~~~~^^
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/support/asynchat.py", line 124, in handle_read
    data = self.recv(self.ac_in_buffer_size)
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/support/asyncore.py", line 377, in recv
    data = self.socket.recv(buffer_size)
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/ssl.py", line 1285, in recv
    return self.read(buflen)
           ~~~~~~~~~^^^^^^^^
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/ssl.py", line 1140, in read
    return self._sslobj.read(len)
           ~~~~~~~~~~~~~~~~~^^^^^
ssl.SSLError: [SYS] unknown error (_ssl.c:2634)

ok

According to @encukou, the failure started to occur after a system update:

• OpenSSL 3.3.0→3.4.0
• Linux 5.15.0→6.8.0
• glibc 2.39→2.40
• GCC 13.2.1→14.2.1
• etc.

Linked PRs

• gh-127257: Add hex code to ssl "unknown error" message #127360
• gh-127257: ssl: Raise OSError for ERR_LIB_SYS #127361
• [3.13] gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) #127812
• [3.12] gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) #127905
• [3.11] gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) (GH-127905) #131970
• [3.10] gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) (GH-127905) #131971

[vstinner]

cc @pablogsal

[vstinner]

Failing tests:

• test_stls (test.test_poplib.TestPOP3Class.test_stls)
• test_stls_context (test.test_poplib.TestPOP3Class.test_stls_context)

stls stands for "Start a TLS session":

    def stls(self, context=None):
        """Start a TLS session on the active connection as specified in RFC 2595.

                context - a ssl.SSLContext
        """

[vstinner]

First failure: https://buildbot.python.org/#/builders/484/builds/6224

That's AMD64 Arch Linux TraceRefs 3.x. test_poplib is also failing on AMD64 Arch Linux Asan 3.x, first failure: https://buildbot.python.org/#/builders/582/builds/6182

[ned-deily]

Are there changes/additions in the OpenSSL 3.4.x branch to its error codes that require updating our header files (see https://github.com/python/cpython/blob/main/Tools/ssl/make_ssl_data.py)?

[encukou]

Thanks for the pointer, @ned-deily! Here's where it led me: #127331

[encukou]

Oh! The [SYS] library means that the error code is a system errno and the message should be retrieved using perror rather than ERR_reason_error_string.

I wonder if this should raise an OSError using PyErr_SetFromErrno, rather than a SSLError.

The failure is an EPIPE, raised when the server is shutting down:

Fatal Python error: Aborted

Current thread 0x00007ff46ce786c0 (most recent call first):
  File "/workspace/Lib/ssl.py", line 1140 in read
  File "/workspace/Lib/ssl.py", line 1285 in recv
  File "/workspace/Lib/test/support/asyncore.py", line 377 in recv
  File "/workspace/Lib/test/test_ftplib.py", line 416 in recv
  File "/workspace/Lib/test/support/asynchat.py", line 124 in handle_read
  File "/workspace/Lib/test/test_poplib.py", line 203 in handle_read
  File "/workspace/Lib/test/support/asyncore.py", line 426 in handle_read_event
  File "/workspace/Lib/test/test_ftplib.py", line 394 in handle_read_event
  File "/workspace/Lib/test/support/asyncore.py", line 89 in read
  File "/workspace/Lib/test/support/asyncore.py", line 156 in poll
  File "/workspace/Lib/test/support/asyncore.py", line 213 in loop
  File "/workspace/Lib/test/test_poplib.py", line 235 in run
  File "/workspace/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/workspace/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007ff46e1c1740 (most recent call first):
  File "/workspace/Lib/threading.py", line 1092 in join
  File "/workspace/Lib/test/test_poplib.py", line 242 in stop
  File "/workspace/Lib/test/test_poplib.py", line 270 in tearDown
  File "/workspace/Lib/unittest/case.py", line 620 in _callTearDown
  File "/workspace/Lib/unittest/case.py", line 663 in run
  File "/workspace/Lib/unittest/case.py", line 716 in __call__
  File "/workspace/Lib/unittest/suite.py", line 122 in run
  File "/workspace/Lib/unittest/suite.py", line 84 in __call__
  File "/workspace/Lib/unittest/suite.py", line 122 in run
  File "/workspace/Lib/unittest/suite.py", line 84 in __call__
  File "/workspace/Lib/unittest/runner.py", line 240 in run
  File "/workspace/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/workspace/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/workspace/Lib/test/libregrtest/single.py", line 135 in test_func
  File "/workspace/Lib/test/libregrtest/single.py", line 91 in regrtest_runner
  File "/workspace/Lib/test/libregrtest/single.py", line 138 in _load_run_test
  File "/workspace/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exc
  File "/workspace/Lib/test/libregrtest/single.py", line 281 in _runtest
  File "/workspace/Lib/test/libregrtest/single.py", line 307 in run_single_test
  File "/workspace/Lib/test/libregrtest/main.py", line 365 in run_test
  File "/workspace/Lib/test/libregrtest/main.py", line 399 in run_tests_sequentially
  File "/workspace/Lib/test/libregrtest/main.py", line 545 in _run_tests
  File "/workspace/Lib/test/libregrtest/main.py", line 580 in run_tests
  File "/workspace/Lib/test/libregrtest/main.py", line 752 in main
  File "/workspace/Lib/test/libregrtest/main.py", line 760 in main
  File "/workspace/Lib/test/__main__.py", line 2 in <module>
  File "/workspace/Lib/runpy.py", line 88 in _run_code
  File "/workspace/Lib/runpy.py", line 198 in _run_module_as_main
Aborted (core dumped)