gh-91539: improve performance of get_proxies_environment

[eendebakpt]

Improve performance of get_proxies_environment when there are many environment variables. Improvements depend on the number of environment variables, but the method is several times faster in this PR.

Fixes#91539

Performance test details

Results:

0.1634683609008789
0.024187326431274414

with

import os
import time
import urllib.request
if 0:
os.environ={ f'{ii}': 1 for ii in range(8000)}
os.environ.update({ f'{ii}_proxy': 1 for ii in range(30)})
def getproxies_environment():
"""Return a dictionary of scheme -> proxy server URL mappings.
Scan the environment for variables named <scheme>_proxy;
this seems to be the standard convention. If you need a
different way, you can pass a proxies dictionary to the
[Fancy]URLopener constructor.
"""
# in order to prefer lowercase variables, process environment in
# two passes: first matches any, second pass matches lowercase only
# select only environment variables which end in (after making lowercase) _proxy candidate_names = [name for name in os.environ.keys() if len(name)>5 and name[-6]=='_'] # fast selection of candidates
environment = [(name, os.environ[name], name.lower()) for name in candidate_names if name[-6:].lower()=='_proxy'] proxies = {}
for name, value, name_lower in environment:
if value and name_lower[-6:] == '_proxy':
proxies[name_lower[:-6]] = value
# CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY
# (non-all-lowercase) as it may be set from the web server by a "Proxy:"
# header from the client
# If "proxy" is lowercase, it will still be used thanks to the next block
if 'REQUEST_METHOD' in os.environ:
proxies.pop('http', None)
for name, value, name_lower in environment:
if name[-6:] == '_proxy':
if value:
proxies[name_lower[:-6]] = value
else:
proxies.pop(name_lower[:-6], None)
return proxies
nn=400
t0=time.time()
for ii in range(nn):
urllib.request.getproxies_environment()
dt=time.time()-t0
print(dt)
t0=time.time()
for ii in range(nn):
getproxies_environment()
dt=time.time()-t0
print(dt)

• Issue: speed up urllib.request.getproxies_environment #91539

[eendebakpt]

@carljm Thanks for the suggestion. Benchmarks show it is just as fast, and much cleaner code.

[carljm]

Sorry I missed this in first review, but the name of the function has a typo (extra underscore) in the NEWS entry.

Also suggested an added comment. (Wouldn't have bothered with this if it was the only thing, but may be worth it if you are doing one more update to fix NEWS anyway.)

Thanks for the improvements to this function!

[eendebakpt]

@ambv As the latest core dev touching Lib/urllib/request.py, would you be able to review this PR?

[orsenthil]

LGTM

[miss-islington]

Thanks @eendebakpt for the PR, and @orsenthil for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

[bedevere-bot]

GH-97918 is a backport of this pull request to the 3.11 branch.

[bedevere-bot]

GH-97919 is a backport of this pull request to the 3.10 branch.