Fix download of Windows Python toolchain on Linux

[jesseschalken]

PR Checklist

Please check if your PR fulfills the following requirements:

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature (please, look at the "Scope of the project" section in the README.md file)
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• Other... Please describe:

What is the current behavior?

1. python_repository rule crashes with: Failed to make interpreter installation read-only. 'chmod' error msg: chmod: cannot access 'lib': No such file or directory because the lib directory is actually called Lib
2. The glob in the generated repo doesn't include the Lib directory causing no builtin libraries to be found when running the cross compiled py_binary.

Issue Number: N/A

What is the new behavior?

1. python_repository rule implementation doesn't crash.
2. Lib directory is included correctly.

Does this PR introduce a breaking change?

• Yes
• No

Other information

[jesseschalken]

I may need some help with the CI failure. The error is from build-runfiles on macOS:

mkdir 'python_x86_64-apple-darwin/lib': File exists

I don't see how this PR could have caused this, and I don't have a mac to test it on.

[UebelAndre]

I may need some help with the CI failure. The error is from build-runfiles on macOS:

mkdir 'python_x86_64-apple-darwin/lib': File exists

I don't see how this PR could have caused this, and I don't have a mac to test it on.

Perhaps this is a flake? But MacOS is a case-insensitive filesystem by default so it can't distinguish the difference between lib and Lib. I feel like the core issue here is that the rule assumes the existence of a path based on the host OS. When instead Lib/lib should be explicitly determined by the platform attribute.

[jesseschalken]

When instead Lib/lib should be explicitly determined by the platform attribute.

Isn't that what this PR does? https://github.com/bazelbuild/rules_python/pull/769/files#diff-2772f4c795e29614d2ca3c7f3302d558abe63bae2379d65285bcdf12e4ea469fR112

[UebelAndre]

When instead Lib/lib should be explicitly determined by the platform attribute.

Isn't that what this PR does? https://github.com/bazelbuild/rules_python/pull/769/files#diff-2772f4c795e29614d2ca3c7f3302d558abe63bae2379d65285bcdf12e4ea469fR112

Ah yes, I misread the lines changed. Would you be willing to rebase and see if the failure is a flake?

[UebelAndre]

I think I found the issue and it's that if you glob for Lib and lib on a case-insensitive platform then both get included and you fail to create runfiles directories. Does something like the following patch help?

diff --git a/python/repositories.bzl b/python/repositories.bzl
index 8bb8a9c..a1fd4cd 100644
--- a/python/repositories.bzl+++ b/python/repositories.bzl@@ -131,6 +131,29 @@ def _python_repository_impl(rctx):
python_bin = "python.exe" if ("windows" in platform) else "bin/python3"
+ if "windows" in platform:+ glob_include = [+ "*.exe",+ "*.dll",+ "bin/**",+ "DLLs/**",+ "extensions/**",+ "include/**",+ "Lib/**",+ "libs/**",+ "Scripts/**",+ "share/**",+ ]+ else:+ glob_include = [+ "bin/**",+ "extensions/**",+ "include/**",+ "lib/**",+ "libs/**",+ "share/**",+ ]+
build_content = """\
# Generated by python/repositories.bzl
@@ -141,18 +164,7 @@ package(default_visibility = ["//visibility:public"])
filegroup(
name = "files",
srcs = glob(
- include = [- "*.exe",- "*.dll",- "bin/**",- "DLLs/**",- "extensions/**",- "include/**",- "lib/**",- "libs/**",- "Scripts/**",- "share/**",- ],+ include = {glob_include},
# Platform-agnostic filegroup can't match on all patterns.
allow_empty = True,
exclude = [
@@ -206,6 +218,7 @@ py_runtime_pair(
py3_runtime = ":py3_runtime",
)
""".format(
+ glob_include = json.encode(glob_include),
python_path = python_bin,
python_version = python_short_version,
)

[jesseschalken]

@UebelAndre Good spot. That fixed it. Thanks!

[UebelAndre]

@groodt Do you have time to take a look at this one this week as well?

[groodt]

LGTM