test_ssl fails on recent FreeBSD (OpenSSL 1.1.1p), minimum_version=TLSv1_2

[vstinner]

test_ssl started to fail today on AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/483/builds/2643

It seems like ssl.SSLContext.minimum_version is now TLSVersion.TLSv1_2.

0:42:57 load avg: 0.32 Re-running test_ssl in verbose mode (matching: test_openssl111_deprecations, test__create_stdlib_context, test_constructor, test_protocol_tlsv1_2, test_min_max_version)
test_openssl111_deprecations (test.test_ssl.BasicSocketTests.test_openssl111_deprecations) ... 
  test_openssl111_deprecations (test.test_ssl.BasicSocketTests.test_openssl111_deprecations) (protocol=<_SSLMethod.PROTOCOL_TLSv1: 3>) ... ERROR
test__create_stdlib_context (test.test_ssl.ContextTests.test__create_stdlib_context) ... ERROR
test_constructor (test.test_ssl.ContextTests.test_constructor) ... ERROR
test_min_max_version (test.test_ssl.ContextTests.test_min_max_version) ... FAIL
test_protocol_tlsv1_2 (test.test_ssl.ThreadedTests.test_protocol_tlsv1_2)
Connecting to a TLSv1.2 server with various client options. ... ERROR

Stdout:

 PROTOCOL_TLSv1_2->PROTOCOL_TLSv1_2 CERT_NONE
 {PROTOCOL_TLS->PROTOCOL_TLSv1_2} CERT_NONE
 PROTOCOL_TLSv1_2->PROTOCOL_TLS CERT_NONE
 {PROTOCOL_TLSv1->PROTOCOL_TLSv1_2} CERT_NONE

======================================================================
ERROR: test_openssl111_deprecations (test.test_ssl.BasicSocketTests.test_openssl111_deprecations) (protocol=<_SSLMethod.PROTOCOL_TLSv1: 3>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 621, in test_openssl111_deprecations
    ssl.SSLContext(protocol)
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/ssl.py", line 500, in __new__
    self = _SSLContext.__new__(cls, protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid or unsupported protocol version 3

======================================================================
ERROR: test__create_stdlib_context (test.test_ssl.ContextTests.test__create_stdlib_context)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1720, in test__create_stdlib_context
    ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/ssl.py", line 810, in _create_unverified_context
    context = SSLContext(protocol)
              ^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/ssl.py", line 500, in __new__
    self = _SSLContext.__new__(cls, protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid or unsupported protocol version 3

======================================================================
ERROR: test_constructor (test.test_ssl.ContextTests.test_constructor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1143, in test_constructor
    ctx = ssl.SSLContext(protocol)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/ssl.py", line 500, in __new__
    self = _SSLContext.__new__(cls, protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid or unsupported protocol version 3

======================================================================
ERROR: test_protocol_tlsv1_2 (test.test_ssl.ThreadedTests.test_protocol_tlsv1_2)
Connecting to a TLSv1.2 server with various client options.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 257, in wrapper
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 3457, in test_protocol_tlsv1_2
    try_protocol_combo(ssl.PROTOCOL_TLSv1_2, ssl.PROTOCOL_TLSv1, False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 2870, in try_protocol_combo
    client_context = ssl.SSLContext(client_protocol)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/ssl.py", line 500, in __new__
    self = _SSLContext.__new__(cls, protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid or unsupported protocol version 3

Stdout:

 PROTOCOL_TLSv1_2->PROTOCOL_TLSv1_2 CERT_NONE
 {PROTOCOL_TLS->PROTOCOL_TLSv1_2} CERT_NONE
 PROTOCOL_TLSv1_2->PROTOCOL_TLS CERT_NONE
 {PROTOCOL_TLSv1->PROTOCOL_TLSv1_2} CERT_NONE

======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests.test_min_max_version)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/warnings_helper.py", line 57, in wrapper
    return test(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1287, in test_min_max_version
    self.assertIn(
    ^^^^^^^^^^^^^^
AssertionError: <TLSVersion.TLSv1_1: 770> not found in {<TLSVersion.SSLv3: 768>, <TLSVersion.TLSv1: 769>}

----------------------------------------------------------------------
Ran 5 tests in 0.113s

FAILED (failures=1, errors=4)
test test_ssl failed

test.pythoninfo:

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 1.1.1p  21 Jun 2022
ssl.OPENSSL_VERSION_INFO: (1, 1, 1, 16, 15)
ssl.OP_ALL: 0x80000054
ssl.OP_NO_TLSv1_1: 0x10000000
ssl.SSLContext.maximum_version: -1
ssl.SSLContext.minimum_version: 771
ssl.SSLContext.options: 2186412116
ssl.SSLContext.protocol: 16
ssl.SSLContext.verify_mode: 2

ssl.default_https_context.maximum_version: -1
ssl.default_https_context.minimum_version: 771
ssl.default_https_context.options: 2186412116
ssl.default_https_context.protocol: 16
ssl.default_https_context.verify_mode: 2

ssl.stdlib_context.maximum_version: -1
ssl.stdlib_context.minimum_version: 771
ssl.stdlib_context.options: 2186412116
ssl.stdlib_context.protocol: 16
ssl.stdlib_context.verify_mode: 0

[vstinner]

cc @koobs

[koobs]

The defaults of the FreeBSD OpenSSL port/package may have changed (or I inadvertently changed them).

Python tests should either either check for TLS version support in the underlying SSL library before running the tests, or skipping (not FAIL'ing) due to non-presence of the required version for the test to pass, as they cant pass a test for protocol version support that doesn't exist.

Any OpenSSL library for any OS under test/CI for CPython may change support for any protocol version or feature at any time, so its worth leveling up the SSL tests in that regard.

[tiran]

@pablogsal The failing tests on FreeBSD are blocking the release pipeline, https://buildbot.python.org/all/#/release_status I'm working on a fix.