speed up urllib.request.getproxies_environment

in module urllib.request， function getproxies_environment （https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L2493）

iterate over all environment variables twice

I made a test:

create 2500 environment variables
call and timing getproxies_environment()
the result is: getproxies_environment() take more than 50ms, which is a big overhead

so if there are many environment variables(like 10 thousands) in a machine, packages depend on function urllib.request.getproxies_environment (like requests does) shall be very slow

is 10 thousands environment variables reasonable？

yes it is, for example, in k8s cluster，kubelete may add a set of environment variables all Services(could be 10 thousands even 100 thousands), in purpose of service discovery

so would it be ok that getproxies_environment don't iterate over all environment variables?

Linked PRs

gh-91539: Small performance improvement of urrlib.request.getproxies_environment() #108771

@bohea

@bohea The following code reduces the computation time by a factor 3 or more on my system.

def getproxies_environment():
    """Return a dictionary of scheme -> proxy server URL mappings.
    Scan the environment for variables named <scheme>_proxy;
    this seems to be the standard convention.  If you need a
    different way, you can pass a proxies dictionary to the
    [Fancy]URLopener constructor.
    """
    # in order to prefer lowercase variables, process environment in
    # two passes: first matches any, second pass matches lowercase only

    # select only environment variables which end in (after making lowercase) _proxy 
    candidate_names = [name for name in os.environ.keys() if name[-6:]=='_'] # fast selection of candidates
    environment = [(name, os.environ[name], name.lower()) for name in candidate_names if name[-6:].lower()=='_proxy'] 
    
    proxies = {}
    for name, value, name_lower in environment:
        if value and name_lower[-6:] == '_proxy':
            proxies[name_lower[:-6]] = value
    # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
    # (non-all-lowercase) as it may be set from the web server by a "Proxy:"
    # header from the client
    # If "proxy" is lowercase, it will still be used thanks to the next block
    if 'REQUEST_METHOD' in os.environ:
        proxies.pop('http', None)
    for name, value, name_lower in environment:
        if name[-6:] == '_proxy':
            if value:
                proxies[name_lower[:-6]] = value
            else:
                proxies.pop(name_lower[:-6], None)
    return proxies

Does this work for you? If so, then I will make a PR

gh-91539: improve performance of get_proxies_environment (#91566)

pythongh-91539: improve performance of get_proxies_environment (pytho…

gh-91539: improve performance of get_proxies_environment (GH-91566)

Merge branch 'main' into dictwatch

Merge branch 'main' into typewatch

pythongh-91539: improve performance of get_proxies_environment (pytho…

gh-91539: Small performance improvement of urrlib.request.getproxies_…

pythongh-91539: Small performance improvement of urrlib.request.getpr…

Navigation Menu

Repository navigation

speed up urllib.request.getproxies_environment #91539

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

speed up urllib.request.getproxies_environment #91539

Description

Linked PRs

Activity

eendebakpt commented on Apr 15, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions