feat(pypi): first check index contents before downloading metadata about distributions

[aignas]

The overall changes to the architecture are:

• First check which packages are on which index.
• Then write these details as facts for future reuse.
• Then use the index_url_overrides to ensure that we are pulling things from the right index for the packages.
• Then download everything.

Notes on implementation:

• This will pull index contents at most once for each index.
• This will make the initial download times longer, but because we have MODULE.bazel.lock file and the facts written there, it should be OK.
• This allows us to just parse the index_url and extra_index_urls from the lock files and use that without printing any warning messages. If we don't see any regressions in testing, I think this code path could be enabled for everyone by default. So experimental_index_url will no longer be experimental. I think this might have been the last thing holding us from flipping the switch.

Fixes#2632
Fixes#3260

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This experimental pull request introduces a significant change to how Python packages are resolved and downloaded from PyPI. Instead of directly attempting downloads, the system now first queries the PyPI index contents to identify available packages and their precise download URLs. This approach aims to improve reliability, enable cross-compilation by fetching platform-specific wheels, and lay the groundwork for more sophisticated package resolution, especially in multi-index or private index scenarios. The changes also refine the handling of "yanked" packages by allowing a more descriptive status.

Highlights

• Enhanced Yanked Package Handling: The yanked field in package metadata now supports None (not yanked) or a string (yank reason), providing more granular status than a simple boolean.
• Optimized HTML Parsing: The parse_simpleapi_html function has been significantly refactored for improved performance and now supports parsing the main PyPI index page to discover package URLs.
• Pre-download Index Content Check: A new mechanism (_get_index_overrides) was introduced to query PyPI indexes for available packages and their specific URLs before attempting to download, allowing for more intelligent and targeted downloads.
• Refined Download Failure Handling: The allow_fail=True parameter was removed from ctx.download calls, indicating that download failures are now expected to be handled by the pre-check logic rather than being silently allowed.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• python/private/pypi/parse_requirements.bzl
  • Updated the yanked field's default value from False to None in _package_srcs and _add_dists functions.
  • Adjusted logic to check for yanked != None instead of yanked or not yanked to correctly interpret the new None state.
• python/private/pypi/parse_simpleapi_html.bzl
  • Modified parse_simpleapi_html to accept a parse_index boolean parameter, enabling it to parse the main PyPI index page for package URLs.
  • Refactored HTML parsing logic for improved efficiency, replacing content.split with content.find for iteration.
  • Introduced _parse_attrs helper function to efficiently extract HTML attributes.
  • Added _unescape_pypi_html utility to decode HTML entities in attribute values, particularly for data-yanked reasons.
  • Updated the yanked field in returned structs to be str | None instead of bool.
• python/private/pypi/simpleapi_download.bzl
  • Added a new _get_index_overrides function to pre-fetch package availability and URLs from all configured indexes.
  • Modified the simpleapi_download function to utilize _get_index_overrides to determine the correct index URL for each package.
  • Removed allow_fail = True from ctx.download calls, relying on the pre-check logic to prevent unnecessary download attempts.
  • Updated the docstring to reflect the new pre-check and targeted download strategy.
  • Propagated the parse_index parameter to _read_simpleapi and _read_index_result.
• tests/pypi/hub_builder/hub_builder_tests.bzl
  • Updated expected yanked values in test structs from False to None to align with the new yanked field type.
• tests/pypi/parse_requirements/parse_requirements_tests.bzl
  • Updated expected yanked values in test structs from False to None to align with the new yanked field type.
• tests/pypi/parse_simpleapi_html/parse_simpleapi_html_tests.bzl
  • Added new test cases to verify correct parsing of data-yanked attributes, including empty reasons and reasons with HTML entities.
  • Updated assertions to expect yanked as str | None instead of bool.

Activity

• This is an experimental pull request, stacked on fix(pypi): return yank reason from SimpleAPI HTML #3656.
• The author is actively seeking collaboration and feedback, specifically on testing with private indexes, unit test coverage, parallel downloads for index contents, using index-provided URLs for package construction, and caching results.
• The changes are related to existing issues pypi downloads blocked by experimental_downloader_config fail silently #2632 and SimpleAPI download failures are cached and require bazel clean #3260.
• No reviewer comments or approvals have been recorded yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly improves the handling of multiple PyPI indexes by introducing a more efficient method for querying index roots and refactoring the HTML parser to support modern features like yanked packages. A security audit found no medium, high, or critical vulnerabilities, confirming adherence to Bazel's security model and absence of injection or data leakage issues. However, a critical error handling placeholder (fail("TODO")) for unavailable indexes still needs to be addressed.

[aignas]

OK, done - PTAL.

[aignas]

OK, PTAL, I am changing the behaviour in a way that will make it easier to default index_url to pypi.org and parse/override params from the lock files as is asked in #2951. In the next PR I plan to just flip the switch by: defaulting the index-url internally to pypi.org and then parsing the requirements files to get the values. I think it is going to be a good thing for 2.0 and if it is breaking, it should be minimally so. I think we should also rename the experimental_index_url_overrides to index_url_overrides.

Given that there are usages of experimental_index_url in the bazel-central-registry, I think it is best to just make the parameter a noop. All of the values are actually using pypi.org/simple.