Pip --proxy parameter is not been passed to the request.

[junqfisica]

pip version

21.0.1

Python version

3.8

OS

Windows 10 / Linux

Additional information

The problem is easier to see if you are behind a proxy.

Description

Running pip install package --proxy http://my_proxy DO NOT pass the proxies to requests.Session.request(...).

As far as I could check this problem is there since version 20.2.2 at least. It was unnoticed until now because, before urllib update, it didn't check ssl_certificate so it didn't matter if your scheme was either http or https. Therefore, this bug is partially a root cause for the SSLError in recent versions of pip. Since users don't have the ability to pass the proxy to pip directly.

Expected behavior

I would expect that running pip install with --proxy, the proxy set here would be passed to the request, overriding any system configuration.

How to Reproduce

1. Start a fresh venv with pip and setuptools only
2. Then set a working proxy using system var. i.e: set https_proxy: http://your_proxy or export https_proxy: http://your_proxy for Linux.
3. Now run pip passing a fake proxy, i.e: pip install numpy --proxy http://127.0.0.2:80 or pip.main(['install', '--proxy=http://127.0.0.2:80', 'numpy']) for esier debug.
4. You should expect an error when installing, but if you have your proxy set correctly in the system it will NOT fail.

PS: You can also do the inverse process. Set a fake proxy on the system and run pip --proxy using a working one. Now you should get an error.

Output

Code of Conduct

• I agree to follow the PSF Code of Conduct

I have already created a post about it here, including a working solution.

[CrazyBoyFeng]

There is a situation that needs to be discussed:
--proxy is not completely inoperative. It works while system proxy is not set. Now our temporary solution to this bug is like this.

So it seems to be wrong in some if branch of determining to use the system proxy or parameter?
That is why I was checking merge_setting(), although I found nothing there.

[uranusjr]

There’s some discussion here as well, and we independenly reached the same conclusion. You’re on the right direction, but the relevant code is actually in the underlying requests package. Pip can try to overwrite this behaviour, but the solution would be pretty hack-y (basically overwrites the value after requests overwirtes it), a conclusion needs to be reached in requests before pip can decide how best to handle this.

[CrazyBoyFeng]

So this is a bug of requests, and handled by psf/requests#5677 psf/requests#5681 psf/requests#5735

To @junqfisica :
--proxy is passed to requests in session.send(proxies). session is build at:

pip/src/pip/_internal/cli/req_command.py

Lines 109 to 112 in 25114e1

	if options.proxy:
	session.proxies = {
	"http": options.proxy,
	"https": options.proxy,

[junqfisica]

@CrazyBoyFeng You are right about the proxy been passed correctly when the system has no proxy set. However, the solution proposed here is quite useless for many users. If you need to pass a proxy is very likely you need it at the system level too, and as I mentioned before, many developers can't change that in working places environments.

To @CrazyBoyFeng :
This is working as expected:

pip/src/pip/_internal/cli/req_command.py

Lines 109 to 112 in 25114e1

	if options.proxy:
	session.proxies = {
	"http": options.proxy,
	"https": options.proxy,

Your reply made me think more about the problem and here is my latest update about it:

Okay, when the PipSession is created the proxy passed to it is correctly set from pip --proxy, which I will refer to as pip_proxy from now on. So, when PipSession request method is called:

pip/src/pip/_internal/network/session.py

Lines 443 to 449 in 3ab760a

	def request(self, method, url, *args, **kwargs):
	# type: (str, str, *Any, **Any) -> Response
	# Allow setting a default timeout on a session
	kwargs.setdefault("timeout", self.timeout)
	# Dispatch the actual request
	return super().request(method, url, *args, **kwargs)

adding a breakpoint there you can notice that self.proxies=pip_proxy, as expected.
Then the parent call is made:

pip/src/pip/_vendor/requests/sessions.py

Lines 530 to 544 in 3ab760a

	proxies = proxies or {}
	settings = self.merge_environment_settings(
	prep.url, proxies, stream, verify, cert
	)
	# Send the request.
	send_kwargs = {
	'timeout': timeout,
	'allow_redirects': allow_redirects,
	}
	send_kwargs.update(settings)
	resp = self.send(prep, **send_kwargs)
	return resp

If you add a breakpoint at line 532, you can still notice that self.proxies = pip_proxy and local proxies are {}. So, self.merge_environment_settings is passed with proxies={}. This now brings us here:

pip/src/pip/_vendor/requests/sessions.py

Lines 701 to 723 in 3ab760a

	def merge_environment_settings(self, url, proxies, stream, verify, cert):
	"""
	Check the environment and merge it with some settings.
	:rtype: dict
	"""
	# Gather clues from the surrounding environment.
	if self.trust_env:
	# Set environment's proxies.
	no_proxy = proxies.get('no_proxy') if proxies is not None else None
	env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
	for (k, v) in env_proxies.items():
	proxies.setdefault(k, v)
	# Look for requests environment configuration and be compatible
	# with cURL.
	if verify is True or verify is None:
	verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
	os.environ.get('CURL_CA_BUNDLE'))
	# Merge all the kwargs.
	proxies = merge_setting(proxies, self.proxies)
	stream = merge_setting(stream, self.stream)

on line 722 there is the call to merge_setting, and local parameter proxies is now your system_proxy, and merge_setting, gives the preference to proxies instead of self.proxies, causing the trouble.

Below I'm providing is a small script isolating the problem as a showcase:

from collections import OrderedDict, Mapping


def to_key_val_list(value):
    """Take an object and test to see if it can be represented as a
    dictionary. If it can be, return a list of tuples, e.g.,

    ::

        >>> to_key_val_list([('key', 'val')])
        [('key', 'val')]
        >>> to_key_val_list({'key': 'val'})
        [('key', 'val')]
        >>> to_key_val_list('string')
        Traceback (most recent call last):
        ...
        ValueError: cannot encode objects that are not 2-tuples

    :rtype: list
    """
    if value is None:
        return None

    if isinstance(value, (str, bytes, bool, int)):
        raise ValueError('cannot encode objects that are not 2-tuples')

    if isinstance(value, Mapping):
        value = value.items()

    return list(value)


def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
    """Determines appropriate setting for a given request, taking into account
    the explicit setting on that request, and the setting in the session. If a
    setting is a dictionary, they will be merged together using `dict_class`
    """

    if session_setting is None:
        return request_setting

    if request_setting is None:
        return session_setting

    # Bypass if not a dictionary (e.g. verify)
    if not (
            isinstance(session_setting, Mapping) and
            isinstance(request_setting, Mapping)
    ):
        return request_setting

    merged_setting = dict_class(to_key_val_list(session_setting))
    merged_setting.update(to_key_val_list(request_setting))

    # Remove keys that are set to None. Extract keys first to avoid altering
    # the dictionary during iteration.
    none_keys = [k for (k, v) in merged_setting.items() if v is None]
    for key in none_keys:
        del merged_setting[key]

    return merged_setting


pip_proxy = {'http': 'http://127.0.0.2:80', 'https': 'http://127.0.0.2:80'}
system_proxy = {'http': 'http://127.1.1.1:80', 'https': 'https://127.1.1.1:80'}

print(f"Default merge: {merge_setting(request_setting=system_proxy, session_setting=pip_proxy)}")
# or ??
print(f"Swap settings:? {merge_setting(session_setting=system_proxy, request_setting=pip_proxy)}")

system_proxy = {}  # no proxy in the system.
print(f"No system proxy: {merge_setting(request_setting=system_proxy, session_setting=pip_proxy)}")

Output:

Default merge: OrderedDict([('http', 'http://127.1.1.1:80'), ('https', 'https://127.1.1.1:80')])
Swap settings:? OrderedDict([('http', 'http://127.0.0.2:80'), ('https', 'http://127.0.0.2:80')])
No system proxy: OrderedDict([('http', 'http://127.0.0.2:80'), ('https', 'http://127.0.0.2:80')])

So, it seems that request.Session gives priority to request_setting, which in this case is the system settings. This may be related to a bug, as mentioned by @uranusjr here, but I'm not entirely sure about it. Also, I'm not sure if pip has much of a saying here, correct me if I'm wrong.

So, I still think would be a good idea to pip to explicitly pass the proxy to the request , avoiding this confusion.

I would like to reinforce, as a developer behind proxy if you use pip with pip --proxy it's because you can't rely on the system settings solely (the way proxy is fetched from it can change depending on OS, messing with the scheme). In my view, it's quite important that when using pip with proxy, this proxy should have a higher priority.

Ps: In my exemple, I gave different ips to the proxy just to highlight the problem, but usually this is not a real scenario, the critical point there is actually the scheme.

[junqfisica]

Okay, I have another solution for this error.
Line:

pip/src/pip/_vendor/requests/sessions.py

Line 530 in 3ab760a

proxies = proxies or {}

should be replaced by:
proxies = proxies or self.proxies

This also fixes the problem.

[uranusjr]

I think we should wait on psf/requests#5735 first. It would fix this problem for us (and other requests users) if I understand it correctly. We can ask requests to make a new release before we upgrade the bundled requests for the next pip release in April and receive the fix.

[CrazyBoyFeng]

My conclusion is: this is a bug of requests (psf/requests#5677), pip should not deal with it.
pip/src/pip/_vendor/requests/sessions.py this file is belong to the requests library.

mateusduboli has already proposed a patch PR one month ago: psf/requests#5735 , but PSF has not yet approved it.

In that patch, mateusduboli adjusted the order of merge setting, finally make parameters higher priority than environment.

If the PSF does not approve it in the end, or the processing is too slow, maybe we can consider override the default parameter of session, just like your solution before.

[junqfisica]

@CrazyBoyFeng and @uranusjr I wouldn't call it override, since request actually gives you the ability to pass it as a parameter, but yeah I really think this is the best way. Also, I don't think request.Session will entirely fix the problem here, since it's more related to how they fetch proxies from the system env.

Let me try to make my point clear:

request.Session tries to automatically get proxies from the system if no proxy is provided to it, which is great. Unfortunately, this solution is not bulletproof and many users need the ability to bypass it and set the proxy directly in the request. So, basically, what I'm trying to say is that we can't trust the system parameters for setting the proxy.

Also is quite misleading if pip gives you the option to set a proxy and in the end, rely on request.Session to set it. For me is clear that this specific bug is caused because PipSession is not given its proxy to the parent request.Session .

[uranusjr]

For me is clear that this specific bug is caused because PipSession is not given its proxy to the parent request.Session.

pip sets the proxies here:

pip/src/pip/_internal/cli/req_command.py

Lines 108 to 113 in 031c34e

	# Handle configured proxies
	if options.proxy:
	session.proxies = {
	"http": options.proxy,
	"https": options.proxy,
	}

where session.proxies is a documented requests.Session attribute that should be preferred over environment varialbes. If that’s true, PipSession would not need to pass the proxy values on every Session.request() call. Turns out that’s not actually the case, which is a requests bug. The bug can be avoided by PipSession explicitly passing the proxies argument, but calling this a bug is like saying it’s your fault you’re using an API that behaves incorrectly, which is not a correct characterisation IMO.