Be able to use different proxies for different index urls

[stefanoborini]

What's the problem this feature will solve?

Possibly this is already known but I could not find an easy solution.
Currently pip does support the use of proxies, but as far as I understand it will only rely on one proxy, and use it for every access. The following scenario would therefore introduce problems:

• a company has both an internal pypi server for its own internal packages, and uses external access to the official pypi.
• the company has set up a proxy. All connections to external urls go through the proxy, but all connections to internal ones must not (or you get an error).

Now, when the configuration is done for the browser, this is not a problem, as the automatic setup uses a proxy.pac script that is able to use logic that maps the requested URL to the proxy server to use (or when not to use it). pip however cannot take advantage of this, and if you have the above situation, you can either connect to the local pypi but not the external one (without proxy setup) or only to the external one but not the local one (with proxy setup)

Describe the solution you'd like

pip already supports a --proxy option. The following solutions could be implemented:

1. extend the grammar of this option to support multiple entries that are mapped to index and extra index, in order.
2. consider --proxy as the default and to extend the --index-url or --extra-index-url so that both the url and the optional proxy can be passed in some form
3. add specific proxy options --index-proxy-url and --extra-index-proxy-url. Given that it's possible to specify multiple --extra-index-url entries, proper mapping must be done.

Alternative Solutions

Unsure. It might be that there's a smarter solution that does not involve pip.

Additional context

I am unsure how the proxy setup scenario given above is prevalent out there. I am sure however that at the moment I am experiencing it.

[cjc7373]

Here's a workaround without involving pip.
There are such tools like clash that can help you do exactly what you want.
You can check out its repo for more information, especially the rules part.

[sethraymond]

I have this exact same need with my work setup. Installing third-party tools isn't really a viable solution for us, so having this capability baked-in would be fantastic!

[uranusjr]

To be honest, I don’t really see the need of a pip feature here, or even a third-party tool. This can be easily automated with a script, each call installing different packages from different indexes, setting the appropriate proxy for each call.

[sethraymond]

Writing a custom script isn't a particularly elegant solution, IMO. Rather, I think having the --proxy option for requirements files is an intuitive solution. Let the proxy be scoped to just the file it's introduced in. For example:

requirements_proxy.txt - this file might look to the official PyPi server, and needs the proxy

--proxy <myproxy>
<package1>
<package2>

requirements_no_proxy.txt - this file is behind the company firewall, so we don't use the proxy

--index-url <local_pypi>
<package3>
<package4>

requirements.txt

-r requirements_no_proxy.txt
-r requirements_proxy.txt

Now, we simply run pip install -r requirements.txt.

[stefanoborini]

To be honest, I don’t really see the need of a pip feature here, or even a third-party tool. This can be easily automated with a script, each call installing different packages from different indexes, setting the appropriate proxy for each call.

You can't do that if you rely on pip resolving dependencies across indexes, and the indexes need different proxies. This is typically the case when one of your internal dependencies A deployed in your internal corporate pypi depends on a package B that is on pypi. The invocation to retrieve A will try to retrieve B, but pypi can't be reached because the proxy setting are configured to reach your internal server, and you can't switch the server middleway though the resolve and retrieve cycle.

[pfmoore]

The stated scenario can be handled using the https_proxy and no_proxy environment variables, I believe (put the internal locations in no_proxy). That should be sufficient - it may be a little clumsy, but honestly, that's a consequence of the company proxying policy. And you can probably write someothing that parses the pac file to automate some of this process.

I consider this to be too much of a niche case to be worth supporting in pip. If anything, I would rather see pip simplify its current network handling options, maybe delegating the complexity of handling proxies, etc, to a library (either requests, or a wrapper around requests) that can be configured via a library-defined config file. Then pip can simply reuse this. All of this sort of network configuration could then be handled by that library.

Adding yet more complexity to pip's network support simply makes it harder for the wider ecosystem to grow, because "only pip supports my network setup" becomes a blocker for the development of other tools.

[stefanoborini]

The stated scenario can be handled using the https_proxy and no_proxy environment variables, I believe (put the internal locations in no_proxy). That should be sufficient - it may be a little clumsy

You can't. When you start pip, it starts connecting with whatever has been passed. You can't change the environment variable while pip is executing, and especially you can't change it according to where it is connecting. This is not process spawning works. Period.

but honestly, that's a consequence of the company proxying policy. And you can probably write someothing that parses the pac file to automate some of this process.

Are you suggesting that I go out to a fortune 500 company and force them to change their proxy policy just because I can't use pip to manage to switch proxy while executing?

I consider this to be too much of a niche case to be worth supporting in pip. If anything, I would rather see pip simplify its current network handling options, maybe delegating the complexity of handling proxies, etc, to a library (either requests, or a wrapper around requests) that can be configured via a library-defined config file. Then pip can simply reuse this

This is already the case, pip must provide the interface to define this. Otherwise what are you suggesting? that I monkeypatch the code?

Adding yet more complexity to pip's network support simply makes it harder for the wider ecosystem to grow, because "only pip supports my network setup" becomes a blocker for the development of other tools.

Pip does not work appropriately in a corporate environment. I'd say it's a pretty major issue.

[uranusjr]

Are you suggesting that I go out to a fortune 500 company and force them to change their proxy policy just because I can't use pip to manage to switch proxy while executing?

I’d suggest going out and recommend a Fortune 500 company to offer a little resource and develop a solution (e.g. what @pfmoore suggested), rather than relying on a volunteer-run freebie project to solve the problem for you.

Sorry for being blunt, but surely you can see a bit of irony in the situation here 🙂