Python unable to see CA certificates in Fedora/Rocky/Alma/RHEL systems

[nonell-viavi]

Hi,

I think there is an issue when attempting to use this precompiled binaries on a rhel-based system.
I've read the quirks section, and I haven't seen a reference to this problem, that is why I'm raising an issue.

The reason we care about this funcionality is because we're trying to use this python + poetry to get "portable" dev environments accross a bunch of developers (using mise).

Issue description

The following code reproduces the problem on a rhel-based system:

docker run -it --rm rockylinux:8 bash -c \
  "dnf update -y && dnf install -y wget ca-certificates && \
  wget https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  tar -xzf cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  ./python/bin/python -c \"import urllib.request; f = urllib.request.urlopen('https://www.python.org'); print(f.read(100))\""

Logs

Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27409290 (26M) [application/octet-stream]
Saving to: 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz'

cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-in 100%[=====================================================================================================================>]  26.14M  8.59MB/s    in 3.0s

2024-05-06 08:57:30 (8.59 MB/s) - 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz' saved [27409290/27409290]

Traceback (most recent call last):
  File "/python/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/python/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/python/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/python/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/python/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/python/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/python/lib/python3.10/http/client.py", line 1455, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/python/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/python/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/python/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/python/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/python/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/python/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/python/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/python/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/python/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>

But the same python program, urlopen works just fine on a debian-based system.

docker run -it --rm debian:12 bash -c \
  "apt update -y && apt install -y wget ca-certificates && \
  wget https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  tar -xzf cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz && \
  ./python/bin/python -c \"import urllib.request; f = urllib.request.urlopen('https://www.python.org'); print(f.read(100))\""

Logs

...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27409290 (26M) [application/octet-stream]
Saving to: 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz'

cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-in 100%[=====================================================================================================================>]  26.14M  8.63MB/s    in 3.0s

2024-05-06 08:53:54 (8.63 MB/s) - 'cpython-3.10.13+20240224-x86_64-unknown-linux-gnu-install_only.tar.gz' saved [27409290/27409290]

b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xed}\xebr\x1bG\xb2\xe6\x7fE\xcc;\x94\xa1X\x91\x1a\xab\x01\x82\xe0M\x14\t\x8f$S2=\xba\xd0\xa6d\x9d\xb3>\x0eE\xa3\xd1\x00\x9a\x04\xba\xa1\xbe\x90\x84g\xe6\x01\xce\x9f}\x85}\xc5}\x84\xfd2\xab\xaa\xbb\xfa\n@\xe4\xe8\xd8\x11\x9a\t\x8b@\xa3\xba.Yy\xaf\xcc\xac\xa3o\x86\x81\x13/\xe6'

Other things I've tried

I've tried installing pip-system-certs and certifi, using the provided pip, but neither of them fixed the issue.

Thank you for reading this far, and let me know if there is something else I can do to help. I'm stuck and unable to fix this by myself (I think).

[dirkpetersen]

I ran into the exact same thing on Python 3.10 (3.8 worked fine) , and used monkey patching in the sitecustomize.py to enable urllib ssl globally per ChatGPT recommendation:

vi $(python -c "import site; print(site.getsitepackages()[0])")/sitecustomize.py

paste this

import ssl
try:
    import certifi
    # Define a function that returns a default SSL context with certifi's CA bundle
    def create_certifi_context(purpose=ssl.Purpose.SERVER_AUTH, *, cafile=None, capath=None, cadata=None):
        return ssl.create_default_context(purpose, cafile=certifi.where(), capath=capath, cadata=cadata)
    # Set the default SSL context creation function to use certifi's CA bundle
    ssl._create_default_https_context = create_certifi_context
except:
    print('certifi package not installed')

[keichi]

I also ran into this. Explicitly specifying the SSL certificate file seems to fix the issue:
export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt

[zanieb]

Is this a problem of the standalone builds? i.e., is this reproducible with a local build with pyenv? I wonder if this is just a difference in how the certificates are configured on the systems?

[keichi]

Just built CPython 3.11.9 with pyenv and tested but this issue didn't happen.

After some digging, I've found that RHEL-based distros store their root CA certificate bundle as /etc/ssl/certs/ca-bundle.crt while Debian-based distros store it as/etc/ssl/certs/ca-certificates.crt. Also these paths seem to be baked in when OpenSSL is built. So yes, this comes from the different CA bundle paths between distros.