Upgrade truststore to 0.9.2

[sethmlarson]

Closes #12892 Thanks @mayeut for reporting and @ichard26 for following this one all the way to the end :)

[notatallshaw]

Btw, just so I understand the user experience, does pip catch ImportError when importing truststore and fallback or give the user a helpful message on resolving or does the user just get the full traceback from truststore?

[sethmlarson]

@notatallshaw Great question, pip catches the ImportError on the initial import of truststore and then falls back to Requests' default behavior which is only certifi: https://github.com/pypa/pip/blob/main/src/pip/_internal/cli/index_command.py#L40

[ichard26]

Thank you!

[ichard26]

It would be nice though if @mayeut could confirm that pip gracefully falls back to certifi only on GraalPy now. I don't happen to have a GraalPy installation around.

[mayeut]

Tested this from a manylinux container and it works.
However I'm seeing 2 warnings WARNING: Disabling truststore because platform isn't supported that are not there on Python 3.8 (well I expected different warnings but still the same kind of UX):

docker run -it --rm quay.io/pypa/manylinux2014_aarch64:2024.08.10-1
[root@97dfc8adf9d4 ~]# manylinux-interpreters ensure graalpy310-graalpy240_310_native
...
[root@97dfc8adf9d4 ~]# cpython3.10 -m pip wheel --no-deps -w . git+https://github.com/sethmlarson/pip.git@truststore-0.9.2
...
[root@97dfc8adf9d4 ~]# graalpy3.10 -m pip install pip-24.3.dev0-py3-none-any.whl
...
[root@97dfc8adf9d4 ~]# graalpy3.10 -m pip -V
<frozen graalpy.pip_hook>:48: RuntimeWarning: You are using an untested version of pip. GraalPy provides patches and workarounds for a number of packages when used with compatible pip versions. We recommend to stick with the pip version that ships with this version of GraalPy.
pip 24.3.dev0 from /opt/_internal/graalpy310-graalpy240_310_native/lib/python3.10/site-packages/pip (python 3.10)
[root@97dfc8adf9d4 ~]# graalpy3.10 -m pip install zipp
<frozen graalpy.pip_hook>:48: RuntimeWarning: You are using an untested version of pip. GraalPy provides patches and workarounds for a number of packages when used with compatible pip versions. We recommend to stick with the pip version that ships with this version of GraalPy.
WARNING: Disabling truststore because platform isn't supported
Collecting zipp
  Downloading zipp-3.20.0-py3-none-any.whl.metadata (3.6 kB)
Downloading zipp-3.20.0-py3-none-any.whl (9.4 kB)
Installing collected packages: zipp
Successfully installed zipp-3.20.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
WARNING: Disabling truststore because platform isn't supported
[root@97dfc8adf9d4 ~]# cpython3.8 -m pip install pip-24.3.dev0-py3-none-any.whl 
...
[root@97dfc8adf9d4 ~]# cpython3.8 -m pip -V
pip 24.3.dev0 from /opt/_internal/cpython-3.8.19/lib/python3.8/site-packages/pip (python 3.8)
[root@97dfc8adf9d4 ~]# cpython3.8 -m pip install pybase64
Collecting pybase64
  Downloading pybase64-1.4.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (8.1 kB)
Downloading pybase64-1.4.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (58 kB)
Installing collected packages: pybase64
  WARNING: The script pybase64 is installed in '/opt/_internal/cpython-3.8.19/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pybase64-1.4.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[mayeut]

The difference in UX comes from

pip/src/pip/_internal/cli/index_command.py

Lines 28 to 47 in 81041f7

	def _create_truststore_ssl_context() -> Optional["SSLContext"]:
	if sys.version_info < (3, 10):
	logger.debug("Disabling truststore because Python version isn't 3.10+")
	return None
	try:
	import ssl
	except ImportError:
	logger.warning("Disabling truststore since ssl support is missing")
	return None
	try:
	from pip._vendor import truststore
	except ImportError:
	logger.warning("Disabling truststore because platform isn't supported")
	return None
	ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
	ctx.load_verify_locations(certifi.where())
	return ctx

Unless adding specific code for graalpy there's nothing much that can be done (we still want the warning if something goes wrong on other interpreters). We're in a bit of a gray area between Python 3.10 & Python 3.13.

[ichard26]

*sigh* alright. The duplicate warning is due to the second PipSession being constructed during the pip self version check. IMHO that warning should be suppressed or downgraded to a debug message.

[sbidoul]

Is this ready to go?

[ichard26]

Is this ready to go?

If we consider the potentially noisy "Disabling truststore because platform isn't supported" warning acceptable. Frankly, I have no strong opinions. I'm of the opinion that it should be downgraded to VERBOSE (or DEBUG) as in most situations, the fact truststore is unsupported is "fine" and not something users will notice.. while they will notice the extra and often duplicated warning.

[sbidoul]

I'm of the opinion that it should be downgraded to VERBOSE (or DEBUG) as in most situations, the fact truststore is unsupported is "fine" and not something users will notice.

Although currently the way to explicitly enable the certifi code path is via --use-deprecated=legacy-certs, hinting at the fact it's a feature that will be removed in the future. So it might be better to keep a warning so affected user know there is something they need to address?

Oh, but I now notice that --use-deprecated=legacy-certs is documented but does not actually work?

option --use-deprecated: invalid choice: 'legacy-certs' (choose from 'legacy-resolver')

update: sorry, PEBKAC

[ichard26]

Although currently the way to explicitly enable the certifi code path is via --use-deprecated=legacy-certs, hinting at the fact it's a feature that will be removed in the future. So it might be better to keep a warning so affected user know there is something they need to address?

What can the user do though?

At some point, pip may transition to only using truststore and falling back to certifi only when it's utterly unavailable, but I doubt we'll dump certifi entirely. Not only is it impossible until we require Python 3.10+ but there are going to be environments that are essentially unfixable (unless truststore is improved to work on everything including a toaster :P)

I don't like low-value, high noise warnings. Ideally, we'd only raise this warning when the user would likely benefit from truststore, like if they're using a custom index or a proxy, but that is way too much work.

Anyway, there isn't much we can do to improve this other than to get rid of the duplicated warning, but that isn't something I'm going to take on myself.

[sbidoul]

At some point, pip may transition to only using truststore and falling back to certifi only when it's utterly unavailable, but I doubt we'll dump certifi entirely. Not only is it impossible until we require Python 3.10+ but there are going to be environments that are essentially unfixable (unless truststore is improved to work on everything including a toaster :P)

That makes sense, but then if certify is here to stay, don't we need a proper option to select the certificate backend, instead of --use-deprecated ?