Skip to content

fix(pypi): make the URL/filename extraction from requirement more robust - #2871

Merged
rickeylev merged 4 commits into
bazel-contrib:mainfrom
aignas:fix/2363/improve-url-parsing
May 12, 2025
Merged

fix(pypi): make the URL/filename extraction from requirement more robust#2871
rickeylev merged 4 commits into
bazel-contrib:mainfrom
aignas:fix/2363/improve-url-parsing

Conversation

@aignas

@aignasaignas commented May 11, 2025

Copy link
Copy Markdown
Collaborator

Summary:

  • Make the requirement line the same as the one that is used in whls. It
    only contains extras and the version if it is present.
  • Add debug log statements if we fail to get the version from a direct
    URL reference.
  • Move some tests from parse_requirements_tests to
    index_sources_tests to improve test
    maintenance.
  • Replace the URL encoded + to a regular + in the filename.
  • Correctly handle the case when the =sha256: is used in the URL.

Once this is merged I plan to tackle #2648 by changing the
parse_requirements code to de-duplicate entries returned
by the parse_requirements function.

I cannot think of anything else that we can do for this as of now, so
will mark the associated issue as resolved.

Fixes#2363
Work towards #2648

Summary:
- Make the requirement line the same as the one that is used in whls. It
only contains extras and the version if it is present.
- Add debug log statements if we fail to get the version from a direct
URL reference.
- Move some tests from `parse_requirements_tests` to
`index_sources_tests` to improve test
maintenance.
- Replace the URL encoded `+` to a regular `+` in the filename.
- Correctly handle the case when the `=sha256:` is used in the URL.
I cannot think of anything else that we can do for this as of now, so
will mark the associated issue as resolved.
Fixesbazel-contrib#2363
@rickeylev
rickeylev added this pull request to the merge queueMay 12, 2025
Merged via the queue into bazel-contrib:main with commit c383c3bMay 12, 2025
@aignas
aignas deleted the fix/2363/improve-url-parsing branch May 14, 2025 04:12
aignas added a commit to aignas/rules_python that referenced this pull request Jul 22, 2025
This fixes the subtle bug introduced in bazel-contrib#2871, where we were dropping
the URL from the requirement, because we can download the sdist
directly. We cannot add `--no-index` because sdists in general may
require extra build dependencies and we had already issues previously
(see 0.36 release notes).
Fixesbazel-contrib#2363
aignas added a commit to aignas/rules_python that referenced this pull request Aug 3, 2025
This fixes the subtle bug introduced in bazel-contrib#2871, where we were dropping
the URL from the requirement, because we can download the sdist
directly. We cannot add `--no-index` because sdists in general may
require extra build dependencies and we had already issues previously
(see 0.36 release notes).
Fixesbazel-contrib#2363
aignas added a commit to aignas/rules_python that referenced this pull request Aug 3, 2025
This fixes the subtle bug introduced in bazel-contrib#2871, where we were dropping
the URL from the requirement, because we can download the sdist
directly. We cannot add `--no-index` because sdists in general may
require extra build dependencies and we had already issues previously
(see 0.36 release notes).
Fixesbazel-contrib#2363
aignas added a commit to aignas/rules_python that referenced this pull request Aug 9, 2025
This fixes the subtle bug introduced in bazel-contrib#2871, where we were dropping
the URL from the requirement, because we can download the sdist
directly. We cannot add `--no-index` because sdists in general may
require extra build dependencies and we had already issues previously
(see 0.36 release notes).
Fixesbazel-contrib#2363
github-merge-queueBot pushed a commit that referenced this pull request Aug 11, 2025
…3115)
This fixes the subtle bug introduced in #2871, where we were dropping
the URL from the requirement, because we can download the sdist
directly. We cannot add --no-index because sdists in general may
require extra build dependencies and we had already issues previously
(see 0.36 release notes).
Fixes#2363Fixes#3131
---------
Co-authored-by: Richard Levasseur <richardlev@gmail.com>
aignas added a commit that referenced this pull request Aug 11, 2025
…3115)
This fixes the subtle bug introduced in #2871, where we were dropping
the URL from the requirement, because we can download the sdist
directly. We cannot add --no-index because sdists in general may
require extra build dependencies and we had already issues previously
(see 0.36 release notes).
Fixes#2363Fixes#3131
---------
Co-authored-by: Richard Levasseur <richardlev@gmail.com>
(cherry picked from commit f6dd386)
Cherry-pick notes: adapted the changelog to mention 1.5.2
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[pypi] Support pulling dependencies using direct url specifiers without using an index

2 participants

@aignas@rickeylev