fix(pypi): pull fewer wheels with experimental_index_url by aignas · Pull Request #3058 · bazel-contrib/rules_python

aignas · 2025-07-05T03:20:44Z

Before this we would pull all of the wheels that the user target configuration
would be compatible with and that meant that it was not customizable. This also
meant that there were a lot of footguns in the configuration where the select
statements were not really foolproof.

With this PR we select only those sources that need to be for the declared
configurations.

Freethreaded support should be done by defining extra freethreaded platforms
using the new builder API. It is done as a followup in #3063.

This is also changing the default platforms to be only the fully supported
platforms. This makes the testing easier and avoids us running into
compatibility issues during the rollout.

Work towards #2747
Fixes#2759
Fixes#2849

aignas · 2025-07-05T03:22:23Z

Created this as WIP to signal that I am working on this and to invite feedback. The CI should be mostly happy but there are a few TODOs before we can fully merge and I want to test it a little bit more.

Freethreaded platform definition should be possible.
Probably something else that I need to think of more carefully.

TODO cleanup after this PR is proven to not break things:

Remove complex config_setting generation where we are generating them from whl filenames - it is just too complex to properly maintain and it has not proven to be effective here. In chore(pypi): remove unused config setting code #3065

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

aignas · 2025-07-06T12:46:43Z

The CI failure is a flake.

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

aignas · 2025-07-06T14:12:17Z

This would be ready to be reviewed, but because it is stacked on two PRs, let's get those out of the way first.

Stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747 TODO: - [ ] Fix the remaining expectations in the unit tests. Maybe make the tests less brittle and define platforms for unit testing.

Stacked on bazel-contrib#3058 Since the parent PR is simplifying the code a great deal, we can remove the unused code and its tests. Hopefully this makes the whole codebase easier to contribute to. There is probably more cleanup that can be done, but since the CI is now green, it's a good start.

keith · 2025-07-07T17:01:14Z

I hit this problem with this change (I don't support windows and use requirements_by_platform to exclude it too):

% bazel query 'deps(...)'
INFO: Repository rules_python++python+python_3_13_aarch64-pc-windows-msvc instantiated at:
<builtin>: in <toplevel>
Repository rule python_repository defined at:
/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:271:36: in <toplevel>
ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:76:34: An error occurred during the fetch of repository 'rules_python++python+python_3_13_aarch64-pc-windows-msvc':
Traceback (most recent call last):
File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl", line 76, column 34, in _python_repository_impl
rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python++python+python_3_13_aarch64-pc-windows-msvc//': java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
Loading: 2635 packages loaded

aignas · 2025-07-07T23:48:06Z

Thank you, I'll take a look later.

Marking this as a draft for now.

aignas · 2025-07-08T04:10:17Z

I hit this problem with this change (I don't support windows and use requirements_by_platform to exclude it too):

% bazel query 'deps(...)'
INFO: Repository rules_python++python+python_3_13_aarch64-pc-windows-msvc instantiated at:
<builtin>: in <toplevel>
Repository rule python_repository defined at:
/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:271:36: in <toplevel>
ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl:76:34: An error occurred during the fetch of repository 'rules_python++python+python_3_13_aarch64-pc-windows-msvc':
Traceback (most recent call last):
File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python+/python/private/python_repository.bzl", line 76, column 34, in _python_repository_impl
rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python++python+python_3_13_aarch64-pc-windows-msvc//': java.io.IOException: Error extracting /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862/cpython-3.13.5+20250702-aarch64-pc-windows-msvc-install_only.tar.gz to /home/ubuntu/.cache/bazel/_bazel_ubuntu/480ee760ab026be301955dc1adc19e40/external/rules_python++python+python_3_13_aarch64-pc-windows-msvc/temp6334258738180209862: Prefix "python/install" was given, but not found in the archive. Here are possible prefixes for this archive: "python".
Loading: 2635 packages loaded

@keith, #3070 should fix your issue, I think. The toolchain pulling in query seems to be unrelated to pip integration, so if it works after #3070 and #3058 is merged, I think it is safe to say that the issue is resolved.

DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747

Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747

Before this PR we would be constructing slightly different environments when the `env_marker_setting` is doing it in the analysis phase and when we are doing it in the repo phase due to how the defaults are handled. In this change we simply reuse the same select statements and add an extra helper that is allowing us to process that. Work towards bazel-contrib#2949 Prep for bazel-contrib#3058

This will be needed when we start selecting wheels entirely in the bzlmod extension evaluation phase (bazel-contrib#3058). This adds a few unit tests to just ensure that we conform to the spec even though the code is very simple. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849

Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747

DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747

Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in bazel-contrib#3058 will be easier to override. Work towards bazel-contrib#2747

DO NOT MERGE: stacked on bazel-contrib#3058 This is a continuation of bazel-contrib#3058 where we define freethreaded platforms. They need to be used only for particular python versions so I included an extra marker configuration attribute where we are using pipstar marker evaluation before using the platform. I think this in general will be a useful tool to configure only particular platforms for particular python versions Work towards bazel-contrib#2548, since this shows how we can define custom platforms Work towards bazel-contrib#2747

Before this PR we would be constructing slightly different environments when the `env_marker_setting` is doing it in the analysis phase and when we are doing it in the repo phase due to how the defaults are handled. In this change we simply reuse the same select statements and add an extra helper that is allowing us to process that. Work towards #2949 Prep for #3058 Co-authored-by: Richard Levasseur <richardlev@gmail.com>

This will be needed when we start selecting wheels entirely in the bzlmod extension evaluation phase (bazel-contrib#3058). This adds a few unit tests to just ensure that we conform to the spec even though the code is very simple. Work towards bazel-contrib#2747 Work towards bazel-contrib#2759 Work towards bazel-contrib#2849

This will be needed when we start selecting wheels entirely in the bzlmod extension evaluation phase (#3058). This adds a few unit tests to just ensure that we conform to the spec even though the code is very simple. Work towards #2747 Work towards #2759 Work towards #2849

Before this PR the configuration for platforms would be built non-incrementally, making it harder for users to override particular attributes of the already configured ones. With this PR the new features introduced in #3058 will be easier to override. Work towards #2747

Before this we would pull all of the wheels that the user target configuration would be compatible with and that meant that it was not customizable. This also meant that there were a lot of footguns in the configuration where the select statements were not really foolproof. With this PR we select only those sources that need to be for the declared configurations. Freethreaded support is done by defining extra freethreaded platforms using the new builder API (bazel-contrib#3063). This is also changing the default platforms to be only the fully supported platforms. This makes the testing easier and avoids us running into compatibility issues during the roll out. Work towards bazel-contrib#2747Fixesbazel-contrib#2759Fixesbazel-contrib#2849

…ed in pipstar

aignas force-pushed the exp/pypi-simplify branch from 1d55c50 to c95dce6Compare July 5, 2025 03:21

aignas changed the title ~~next: pypi stabilization of experimental_index_url~~fix(pypi): pull fewer wheels with experimental_index_urlJul 5, 2025

aignas mentioned this pull request Jul 6, 2025
feat: freethreaded support for the builder API #3063
Merged

aignas force-pushed the exp/pypi-simplify branch 2 times, most recently from dc50a12 to 855a673Compare July 6, 2025 13:42

aignas mentioned this pull request Jul 6, 2025
chore(pypi): remove unused config setting code #3065
Merged

rickeylev reviewed Jul 6, 2025
View reviewed changes

aignas marked this pull request as ready for review July 7, 2025 13:15

aignas requested a review from groodt as a code owner July 7, 2025 13:15

This was referenced Jul 7, 2025
experimental_index_url slows down downloads by 25-50% #2849
Closed
"Illegal ambiguous match" error with experimental_index_url#2759
Closed

aignas marked this pull request as draft July 7, 2025 23:50

aignas marked this pull request as ready for review July 8, 2025 04:09

aignas mentioned this pull request Jul 8, 2025
Create a test that tests whl_library with an sdist #3071
Closed

hartikainen reviewed Jul 8, 2025
View reviewed changes

Comment threadpython/private/pypi/select_whl.bzl Outdated

aignas requested a review from hartikainen July 12, 2025 14:19

aignas force-pushed the exp/pypi-simplify branch from 891af92 to 0b8596dCompare August 9, 2025 12:21

aignas marked this pull request as draft August 9, 2025 13:45

aignas force-pushed the exp/pypi-simplify branch 2 times, most recently from c757256 to e2d2dd0Compare August 10, 2025 15:11

rickeylev approved these changes Aug 16, 2025
View reviewed changes

aignas force-pushed the exp/pypi-simplify branch from 294d4b1 to b33d475Compare August 16, 2025 12:12

aignas added 3 commits August 16, 2025 21:13

test: check that requirements files with different versions are handl…
d0ea5fe
…ed in pipstar

fixup tests
f1e9eee

aignas mentioned this pull request Aug 17, 2025
fix(pypi): correctly handle different package versions #3186
Merged

scasagrande mentioned this pull request Sep 8, 2025
version 1.6.0+ with experimental_index_url are fetching incorrect wheels for glibc version #3250
Closed

aignas mentioned this pull request Sep 9, 2025
fix(pypi): select the lowest available libc version by default #3255
Merged

Navigation Menu

Uh oh!

Uh oh!

fix(pypi): pull fewer wheels with experimental_index_url - #3058

fix(pypi): pull fewer wheels with experimental_index_url#3058
aignas merged 3 commits into
bazel-contrib:mainfrom
aignas:exp/pypi-simplify

aignas commented Jul 5, 2025•
edited
Loading

Uh oh!

aignas commented Jul 5, 2025•
edited
Loading

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

keith commented Jul 7, 2025•
edited
Loading

Uh oh!

aignas commented Jul 7, 2025•
edited
Loading

Uh oh!

aignas commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

aignas commented Jul 5, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 5, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

aignas commented Jul 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

keith commented Jul 7, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 7, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aignas commented Jul 5, 2025•
edited
Loading

aignas commented Jul 5, 2025•
edited
Loading

keith commented Jul 7, 2025•
edited
Loading

aignas commented Jul 7, 2025•
edited
Loading