Read index urls from the requirements files and instead have a flag

[aignas]

Summary from #2564

pip.parse(
 experimental_bazel_downloader = True,
 experimental_index_url_overrides = {"package": "https//baz.com/simple", ...},
 requirements_lock = "requirements.txt",
 ...

If requirements.txt has:

--index_url https://foo/simple
--extra-index-url https://bar/simple
--extra-index-url https://foobar/simple

package==1.2.3 \
 --hash=sha256:abcd1234

The experimental_bazel_downloader = True API would be functionally equivalent to:

pip.parse(
 experimental_index_url = "https://foo/simple",
 experimental_extra_index_urls = ["https://bar/simple", "https://foobar/simple"]
 experimental_index_url_overrides = {"package": "https//baz.com/simple", ...}, # note no "oauth2accesstoken"
 requirements_lock = "requirements.txt",
 ...

The net result of such a change is that the requirements.in file is now the canonical source for index URLs no matter how the lock file is made:

flowchart TD
 A[req.in] --> B[pip compile<br>uv pip compile]
 A --> C[compile_pip_requirements]
 B --> D[req.lock]
 C --> D
 D --> |effectively sets| E[pip.parse.experimental_index_url<br>pip.parse.experimental_extra_index_urls]

Loading

Proposal summary:

1. Remove experimental_index_url, experimental_extra_index_urls
2. Add boolean experimental_bazel_downloader (or similar wording)
3. Have pip.parse pull index URLs from the requirements lock file. If not present, default to public PyPI.

I think keeping experimental_index_url_overrides is prudent.

Alternative:

1. Keep experimental_index_url, experimental_extra_index_urls but support special options like DEFER or similar wording that causes pip.parse to pull index URLs from the requirements lock file.

Originally posted by @dougthor42 in #2564 (comment)

[martis42]

If a repository is used in different closed off environments, it is possible that each environment uses a different pypi index address (aka the pypi mirror of this environment). Right now this is trivial to model due to

pip-parse(
 hub_name = "some_pip_hub",
 envsubst = ["PIP_INDEX_URL"],
 experimental_index_url = "${PIP_INDEX_URL:-https://pypi.org/simple}",
)

which allows easily changing which pypi index is used. It would be troublesome in such a case, if the pypi index is hard coded via the lock file.

Or should such use cases be handled in the future via --downlaoder_config? Assuming this is the desired solution, does this work for sdist packages? To my understanding for packages not available as wheel experimental_index_url falls back to python -m pip --isolated wheel .... pip would not know about --downlaoder_config.

[aignas]

With #3657 this will be doable. I expect that we can just parse the index URLs from the requirements files and or args and then use the logic there to use the downloader where we can.

The PR worked for docs building in rules_python but lacks testing in the wild. If you have time, please test and give feedback. Any feedback is welcome.