Pip 25.0 regression: truststore is not used for installing build dependencies

[JustBeefalo]

Description

I am behind a SSL terminating proxy, and have installed my own root CA.
I have a python source project with a standard pyproject.toml which includes dependencies

dependencies = [
    ...
]

After updating from pip 24.3.1 to 25.0, if I perform a pip install ., I see SSLErrors

In the same environment, pip install <dependency> continues to work without any SSL errors, so I'm confident I have installed my root CA correctly. Problem appears isolated to installing dependencies read from a pyproject.toml file.

While pip helpfully tells me note: This error originates from a subprocess, and is likely not a problem with pip., the only change made to my environment is the pip version, and reverting back to 24.3.1 fixes the issue, so I do believe it is a problem with pip.

Expected behavior

Pip should use my system installed root CA for SSL connections while installing dependencies listed in a pyproject.toml.

pip version

25.0

Python version

3.11.11

OS

RHEL 9

How to Reproduce

1. Be behind a proxy that terminates SSL connections and presents a new cert resigned with its own root CA.
2. Verify you've correctly installed your own root CA, and that pip will use it when installing packages (e.g., pip install --upgrade setuptools should return no error).
3. Attempt to build/install a python package with a pyproject.toml that lists dependencies (e.g., pip install .).

Output

$ pip install .
Processing
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Collecting setuptools>=61.0
        WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
        WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
      ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Thanks for reporting, can you also please provide your configuration, e.g. pip config list

[JustBeefalo]

Just in case it helps, and to clarify, I am using setuptools as my build system.

[build-system]                         
requires = ["setuptools>=61.0"]        
build-backend = "setuptools.build_meta"

In both environments where I can and cannot reproduce the issue, the actual installed version of setuptools was unchanged (75.8.0).

Thanks for reporting, can you also please provide your configuration, e.g. pip config list

Returns no output.

$ pip config list

[notatallshaw]

Thanks, it may take a while to debug, do you know how to downgrade pip as a workaround for now?

@ichard26 any thoughts? I assume this is related to passing the cert config to the subprocess, could there be an issue when there is an empty config?

[JustBeefalo]

Yes, I can downgrade to 24.3.1 as a workaround for now.

Thanks for taking the time to look into this.

[ichard26]

Good morning, how are you passing certificate and proxy configuration to pip? Via environment variables? Or have you done nothing special to tell pip about the system certificate and proxy. I'd like to know as pip has different logic when a cert/proxy is explicitly given (--cert, --proxy, or via the corresponding envvars) or implicitly used.

[JustBeefalo]

My proxy settings are set via environment variables (e.g., HTTPS_PROXY, HTTP_PROXY). I'm not doing anything special to tell pip about the cert, rather it's installed via the system wide trust store.

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/securing_networks/using-shared-system-certificates_securing-networks#the-system-wide-trust-store_using-shared-system-certificates

[ichard26]

Hmm, #13063 changed pip to always pass --cert to the build dependency install subprocess, either to pip's CA bundle, or to whatever the user had set. I wonder if that disables truststore (which is what enables system CAs to work with zero-config) in the build environment.

I unfortunately don't have a system CA that I can easily test with, could you try this patch?

diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
index e820dc3d5..9b7582c42 100644
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -246,8 +246,8 @@ class BuildEnvironment:
             # target from config file or env var should be ignored
             "--target",
             "",
-            "--cert",
-            finder.custom_cert or where(),
+            # "--cert",
+            # finder.custom_cert or where(),
         ]
         if logger.getEffectiveLevel() <= logging.DEBUG:
             args.append("-vv")

You can install the modified pip via pip install https://github.com/ichard26/pip/archive/truststore-hotfix.zip.

[JustBeefalo]

Yep, that patch fixes it.

Processing
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done