Allow pipx to use sytem certificates

[BeyondEvil]

Describe the bug
pipx install fails due to corporate network/proxy with self-signed certificate.

From the log:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)'))': /simple/hatch/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)'))': /simple/hatch/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)'))': /simple/hatch/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)'))': /simple/hatch/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)'))': /simple/hatch/
ERROR: Could not find a version that satisfies the requirement hatch (from versions: none)
ERROR: No matching distribution found for hatch

Note that if I create a venv, things work as expected. So the system python/pip handles the (installed) self-signed cert:

syncer on  chore/setup-hatch [!?] 
❯ python3 -m venv .venv                                                             

syncer on  chore/setup-hatch [!?] took 2s 
❯ source .venv/bin/activate

syncer on  chore/setup-hatch [!?] via 🐍 v3.10.6 (.venv) 
❯ pip install hatch    
Collecting hatch
  Downloading hatch-1.6.3-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.0/85.0 KB 2.7 MB/s eta 0:00:00
Collecting platformdirs>=2.5.0
  Downloading platformdirs-3.2.0-py3-none-any.whl (14 kB)

**snip**

Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 14.3 MB/s eta 0:00:00
Using legacy 'setup.py install' for pyperclip, since package 'wheel' is not installed.
Installing collected packages: rfc3986, pyperclip, ptyprocess, distlib, zipp, tomlkit, tomli-w, tomli, sniffio, shellingham, pygments, pycparser, pluggy, platformdirs, pexpect, pathspec, packaging, more-itertools, mdurl, jeepney, idna, h11, filelock, editables, click, certifi, virtualenv, userpath, markdown-it-py, jaraco.classes, importlib-metadata, hyperlink, hatchling, cffi, anyio, rich, httpcore, cryptography, SecretStorage, httpx, keyring, hatch
  Running setup.py install for pyperclip ... done
Successfully installed SecretStorage-3.3.3 anyio-3.6.2 certifi-2022.12.7 cffi-1.15.1 click-8.1.3 cryptography-40.0.1 distlib-0.3.6 editables-0.3 filelock-3.10.7 h11-0.14.0 hatch-1.6.3 hatchling-1.13.0 httpcore-0.16.3 httpx-0.23.3 hyperlink-21.0.0 idna-3.4 importlib-metadata-6.1.0 jaraco.classes-3.2.3 jeepney-0.8.0 keyring-23.13.1 markdown-it-py-2.2.0 mdurl-0.1.2 more-itertools-9.1.0 packaging-23.0 pathspec-0.11.1 pexpect-4.8.0 platformdirs-3.2.0 pluggy-1.0.0 ptyprocess-0.7.0 pycparser-2.21 pygments-2.14.0 pyperclip-1.8.2 rfc3986-1.5.0 rich-13.3.3 shellingham-1.5.0.post1 sniffio-1.3.0 tomli-2.0.1 tomli-w-1.0.0 tomlkit-0.11.7 userpath-1.8.0 virtualenv-20.21.0 zipp-3.15.0

syncer on  chore/setup-hatch [!?] via 🐍 v3.10.6 (.venv) took 17s 
❯ 

I'm using the zipapp (pipx.pyz), version (1.2.0)

My environment is Ubuntu 22.04 running in WSL on Windows 10.

Python is 3.10.6

How to reproduce

syncer on  chore/setup-hatch [!?] 
❯ python3 /home/jim/dev/afa/syncer/.deps/pipx/1.2.0/pipx.pyz --verbose install hatch
usage: pipx.pyz [-h] [--version] {install,uninject,inject,upgrade,upgrade-all,uninstall,uninstall-all,reinstall,reinstall-all,list,run,runpip,ensurepath,environment,completions} ...
pipx.pyz: error: unrecognized arguments: --verbose

To reproduce I guess you have to be in an environment that has a proxy with a self-signed cert.

Expected behavior
pipx successfully installs package.

cmd_2023-03-29_13.44.16.log

[BeyondEvil]

Adding export PIP_CERT="/etc/ssl/certs" seems to have solved it.

However, that seems like a hack.

Happy to help troubleshoot further.

[in03]

I'm having this issue too. I can get around this for standard pip using the pip-system-certs package.

pip install pip-system-certs config --trusted-host pypi.org --trusted-host files.pythonhosted.org

Unfortunately pipx doesn't seem to respect it. I'd try the above workaround but I'm on MacOS and I can't seem to find where it stores its certs.

[gaborbernat]

PR welcome to support such custom flags during package installation.

[Houstonwp]

This can be solved by setting the global cert flag in pip (where ca_certs is the folder containing your custom cert):

pip config set global.cert ~/ca_certs

or by setting the SSL_CERT_FILE environment variable:

export CERT_PATH=/etc/ssl/certs/ZscalerRootCA.pem
export CERT_DIR=/etc/ssl/certs/
export SSL_CERT_FILE=${CERT_PATH} 
export SSL_CERT_DIR=${CERT_DIR}
export REQUESTS_CA_BUNDLE=${CERT_PATH} 

Source: zscaler docs

[BeyondEvil]

This can be solved by setting the global cert flag in pip (where ca_certs is the folder containing your custom cert):

pip config set global.cert ~/ca_certs

or by setting the SSL_CERT_FILE environment variable:

export CERT_PATH=/etc/ssl/certs/ZscalerRootCA.pem
export CERT_DIR=/etc/ssl/certs/
export SSL_CERT_FILE=${CERT_PATH} 
export SSL_CERT_DIR=${CERT_DIR}
export REQUESTS_CA_BUNDLE=${CERT_PATH} 

Source: zscaler docs

While a good solution, isn't it more of a workaround? Shouldn't pipx solve this? At least in my case where pip handled it without any extras.

[gaby]

I spent 3 hrs today debugging why an app suddently stopped working with SSL Cert Errors. Turns out it was because we migrated fron pip to pipx, and it doesnt use the system certificates... 🤦‍♂️

[gaby]

@gaborbernat Setting the ENV's from the zscaler post above makes this work. Is this something pipx could do without adding another flag?

At least on linux the ca-certificates package creates a bundle/certs at /etc/ssl/certs/ca-certificates.crt for file and /etc/ssl/certs/ for dir.

This .crt file will have all the commonly know CA certs + whichever custom ones you added to the system

[gaborbernat]

I am no longer involved in maintaining this project. I moved over to UV tool.