fix(runfiles): correct Python runfiles path assumption by jsing-canva · Pull Request #3086 · bazel-contrib/rules_python

jsing-canva · 2025-07-14T07:25:39Z

The current _FindPythonRunfilesRoot() implementation assumes that
the Python module has been unpacked four levels below the runfiles
directory. This is not the case in multiple situations, for example when
rules_pycross is in use and has installed the module via pypi (in which
case it is five levels below runfiles).

Both strategies already know where the runfiles directory exists -
implement _GetRunfilesDir() on the _DirectoryBased strategy, then call
_GetRunfilesDir() in order to populate self._python_runfiles_dir.

Stop passing a bogus path to runfiles.Create() in testCurrentRepository(),
such that the test actually uses the appropriate runfiles path.

Fixes#3085

aignas · 2025-10-11T04:01:52Z

Hello, going through old PRs and was wondering if you would like to rebase this and see if you can get the CI pass? Recently there were some changes in runfiles to make the compact manifest work, so this may require some work to cleanly rebase.

jsing-canva · 2025-10-23T01:21:54Z

Hello, going through old PRs and was wondering if you would like to rebase this and see if you can get the CI pass? Recently there were some changes in runfiles to make the compact manifest work, so this may require some work to cleanly rebase.

Thanks for the follow up - have rebased.

I had a closer look at the CI failures (Windows) yesterday and will need some guidance on how you want to proceed. The issue is caused by what appears to be a bug in the zip file bootstrap, which is being used by Windows in the two failing tests (//tests/runtime_env_toolchain:bootstrap_script_test, //tests/runtime_env_toolchain:toolchain_runs_test). The issue is caused by RUNFILES_DIR being incorrectly set to the location where the first stage/zip is unpacked, rather than the actual module space (the temporary Bazel.runfiles_*). This results in the following failure:

ValueError: C:\temp\Bazel.runfiles_9i4gshx2\runfiles\rules_python\tests\runtime_env_toolchain\toolchain_runs_test.py does not lie under the runfiles root c:\b\uzvqz6q5\execroot\rules_python\bazel-out\x64_windows-fastbuild-st-ac131bc5af1e\bin\tests\runtime_env_toolchain\bootstrap_script_test.exe.runfiles

Running the same code/test via a zip from a py_binary works correctly (and the other runfiles tests pass). I've not tracked down the exact source of the zip file bootstrap, however it is not part of the rules_python repo and is presumably coming from the toolchain that is being used in the test (commenting out the extra_toolchains part of the tests results in the runfiles code working, then failing due to other reasons, which seems to confirm this).

So given all of this, I'm not sure how we want to proceed... we either need a toolchain that does not have a buggy zip file bootstrap, or we need to skip this particular test (at least the runfiles part of it) on Windows.

Preferences/suggestions?

jsing-canva · 2025-10-23T03:09:26Z

(I've pushed two changes - one that adds an actual Rlocation() data file test to //tests/runfiles and another that adds a workaround for the broken zip bootstrap in //tests/runtime_env_toolchain - hopefully this is acceptable)

jsing-canva · 2025-11-05T08:08:53Z

@aignas any chance we can move this forward?

aignas · 2025-11-05T08:48:55Z

We'll need to fix our CI before we can merge.

jsing-canva · 2025-11-05T12:51:03Z

We'll need to fix our CI before we can merge.

@aignas - #3393 should resolve this (which I've also rebased this change on for now).

The current _FindPythonRunfilesRoot() implementation assumes that the Python module has been unpacked four levels below the runfiles directory. This is not the case in multiple situations, for example when rules_pycross is in use and has installed the module via pypi (in which case it is five levels below runfiles). Both strategies already know where the runfiles directory exists - implement _GetRunfilesDir() on the _DirectoryBased strategy, then call _GetRunfilesDir() in order to populate self._python_runfiles_dir. Stop passing a bogus path to runfiles.Create() in testCurrentRepository(), such that the test actually uses the appropriate runfiles path. Fixesbazel-contrib#3085

jsing-canva · 2025-11-06T03:28:10Z

@aignas rebased and finally got everything to be green again!

phst · 2026-01-18T22:12:54Z

I have a gut feeling that the issue described in #3086 (comment) has caused the failure in https://github.com/phst/rules_elisp/actions/runs/21100412824/job/60683799554?pr=1666:

INFO: From Testing //dev:requirements.test:
Traceback (most recent call last):
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\_main\dev\_requirements.test_stage2_bootstrap.py", line 499, in <module>
main()
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\_main\dev\_requirements.test_stage2_bootstrap.py", line 493, in main
_run_py_path(main_filename, args=sys.argv[1:])
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\_main\dev\_requirements.test_stage2_bootstrap.py", line 287, in _run_py_path
runpy.run_path(main_filename, run_name="__main__")
File "<frozen runpy>", line 287, in run_path
File "<frozen runpy>", line 98, in _run_module_code
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python+\python\private\pypi\dependency_resolver\dependency_resolver.py", line 270, in <module>
main()
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python++config+pypi__click\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python++config+pypi__click\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python++config+pypi__click\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python++config+pypi__click\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python+\python\private\pypi\dependency_resolver\dependency_resolver.py", line 111, in main
resolved_srcs = [_locate(bazel_runfiles, src) for src in srcs]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python+\python\private\pypi\dependency_resolver\dependency_resolver.py", line 82, in _locate
return bazel_runfiles.Rlocation(file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python+\python\runfiles\runfiles.py", line 306, in Rlocation
source_repo = self.CurrentRepository(frame=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_1w2yok13\runfiles\rules_python+\python\runfiles\runfiles.py", line 384, in CurrentRepository
caller_runfiles_path = os.path.relpath(caller_path, self._python_runfiles_root)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen ntpath>", line 807, in relpath
ValueError: path is on mount 'C:', start on mount 'D:'

This test comes from https://github.com/phst/rules_elisp/blob/4ae229b369484a4a764dd7f6d1e30aadce49a7b6/dev/BUILD#L53-L66.

If that's the case, is there a workaround?

aignas · 2026-01-19T03:21:13Z

Could you please create a new issue so that this doesn't get lost?

With any Python version on Windows, `RUNFILES_*` environment variables still point to an intermediate stage path instead of the module path, causing `CurrentRepository()` to raise a `ValueError` since `rules_python` v1.8.0: ``` ValueError: C:\Users\bot\AppData\Local\Temp\Bazel.runfiles_gh34ij5_\runfiles\+_repo_rules+cpython\my_script.py does not lie under the runfiles root C:/cache/ab1cdef2/execroot/_main/bazel-out/x64_windows-fastbuild/bin/external/+_repo_rules+cpython/install.exe.runfiles ``` This was not the case with `rules_python` v1.7.0. The issue stems from bazel-contrib#3086 which, by eliminating a wrong assumption, also brought a stricter behavior. Since bazel-contrib#3086 came up with a corresponding workaround in `//tests/runtime_env_toolchain:toolchain_runs_test`, the proposed fix simply consists in moving it to `CurrentRepository()`, thus adding another case to the workaround introduced by bazel-contrib#1634 for Python < 3.11 - so assuming the main repository on Windows as well. Removing the workaround from `CurrentRepository()`, would make the test fail as follows: ``` ==================== Test output for //tests/runtime_env_toolchain:toolchain_runs_test: E ====================================================================== ERROR: test_ran (__main__.RunTest.test_ran) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\tests\runtime_env_toolchain\toolchain_runs_test.py", line 13, in test_ran settings_path = rf.Rlocation( "rules_python/tests/support/current_build_settings.json" ) File "C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\python\runfiles\runfiles.py", line 306, in Rlocation source_repo = self.CurrentRepository(frame=2) File "C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\python\runfiles\runfiles.py", line 404, in CurrentRepository raise ValueError( ...<3 lines>... ) ValueError: C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\tests\runtime_env_toolchain\toolchain_runs_test.py does not lie under the runfiles root c:\users\user\_bazel_user\cxxeswjo\execroot\_main\bazel-out\x64_windows-fastbuild-st-c530e4918e48\bin\tests\runtime_env_toolchain\toolchain_runs_test.exe.runfiles ```

With any Python version on Windows, `RUNFILES_*` environment variables still point to an intermediate stage path instead of the module path, causing `CurrentRepository()` to raise a `ValueError` since `rules_python` v1.8.0: ``` ValueError: C:\Users\bot\AppData\Local\Temp\Bazel.runfiles_gh34ij5_\runfiles\+_repo_rules+cpython\my_script.py does not lie under the runfiles root C:/cache/ab1cdef2/execroot/_main/bazel-out/x64_windows-fastbuild/bin/external/+_repo_rules+cpython/install.exe.runfiles ``` This was not the case with `rules_python` v1.7.0. The issue stems from bazel-contrib#3086 which, by eliminating a wrong assumption, also brought a stricter behavior. Since bazel-contrib#3086 came up with a corresponding workaround in `//tests/runtime_env_toolchain:toolchain_runs_test`, the proposed fix simply consists in moving it to `CurrentRepository()`, thus adding another case to the workaround introduced by bazel-contrib#1634 for Python < 3.11 - so assuming the main repository on Windows as well. Removing the workaround from `CurrentRepository()`, would make the test fail as follows: ``` ==================== Test output for //tests/runtime_env_toolchain:toolchain_runs_test: E ====================================================================== ERROR: test_ran (__main__.RunTest.test_ran) ---------------------------------------------------------------------- Traceback (most recent call last): [...] ValueError: C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\tests\runtime_env_toolchain\toolchain_runs_test.py does not lie under the runfiles root c:\users\user\_bazel_user\cxxeswjo\execroot\_main\bazel-out\x64_windows-fastbuild-st-c530e4918e48\bin\tests\runtime_env_toolchain\toolchain_runs_test.exe.runfiles ```

With any Python version on Windows, `RUNFILES_*` environment variables still point to an intermediate stage path instead of the module path, causing `CurrentRepository()` to raise a `ValueError` since `rules_python` v1.8.0: ``` ValueError: C:\Users\bot\AppData\Local\Temp\Bazel.runfiles_gh34ij5_\runfiles\+_repo_rules+cpython\my_script.py does not lie under the runfiles root C:/cache/ab1cdef2/execroot/_main/bazel-out/x64_windows-fastbuild/bin/external/+_repo_rules+cpython/install.exe.runfiles ``` This was not the case with `rules_python` v1.7.0. The issue stems from bazel-contrib#3086 which, by eliminating a wrong assumption, also brought a stricter behavior. Since bazel-contrib#3086 came up with a corresponding workaround in `//tests/runtime_env_toolchain:toolchain_runs_test`, the proposed fix simply consists in moving it to `CurrentRepository()`, thus adding another case to the workaround introduced by bazel-contrib#1634 for Python < 3.11. It therefore leads to assuming the main module path on Windows as well. Removing the workaround from `CurrentRepository()` would make the test fail as follows: ``` ==================== Test output for //tests/runtime_env_toolchain:toolchain_runs_test: E ====================================================================== ERROR: test_ran (__main__.RunTest.test_ran) ---------------------------------------------------------------------- Traceback (most recent call last): [...] ValueError: C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\tests\runtime_env_toolchain\toolchain_runs_test.py does not lie under the runfiles root c:\users\user\_bazel_user\cxxeswjo\execroot\_main\bazel-out\x64_windows-fastbuild-st-c530e4918e48\bin\tests\runtime_env_toolchain\toolchain_runs_test.exe.runfiles ```

With any Python version on Windows, `RUNFILES_*` environment variables still point to an intermediate stage path instead of the module path, causing `CurrentRepository()` to raise a `ValueError` since `rules_python` v1.8.0: ``` ValueError: C:\Users\bot\AppData\Local\Temp\Bazel.runfiles_gh34ij5_\runfiles\+_repo_rules+cpython\my_script.py does not lie under the runfiles root C:/cache/ab1cdef2/execroot/_main/bazel-out/x64_windows-fastbuild/bin/external/+_repo_rules+cpython/install.exe.runfiles ``` This was not the case with `rules_python` v1.7.0. The issue stems from bazel-contrib#3086 which, by eliminating a wrong assumption, also brought a stricter behavior. Since bazel-contrib#3086 came up with a corresponding workaround in `//tests/runtime_env_toolchain:toolchain_runs_test`, the proposed fix simply consists in moving it to `CurrentRepository()`, thus adding another case to the workaround introduced by bazel-contrib#1634 for Python < 3.11. It therefore leads to assuming the main module path on Windows as well. Removing the workaround from `CurrentRepository()` would make the test fail as follows: ``` ==================== Test output for //tests/runtime_env_toolchain:toolchain_runs_test: E ====================================================================== ERROR: test_ran (__main__.RunTest.test_ran) ---------------------------------------------------------------------- Traceback (most recent call last): [...] ValueError: C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\tests\runtime_env_toolchain\toolchain_runs_test.py does not lie under the runfiles root c:\users\user\_bazel_user\cxxeswjo\execroot\_main\bazel-out\x64_windows-fastbuild-st-c530e4918e48\bin\tests\runtime_env_toolchain\toolchain_runs_test.exe.runfiles ``` Fixesbazel-contrib#3579.

phst · 2026-02-11T19:54:45Z

Could you please create a new issue so that this doesn't get lost?

Probably #3579 is the corresponding bug

We haven’t isolated bazel-contrib/rules_python#3086 (comment) yet, and bazel-contrib/rules_python#3579 appears to be a separate issue, so it’s better to get back to a known-good state to unblock rules_python updates.

phst · 2026-02-18T21:12:04Z

Could you please create a new issue so that this doesn't get lost?

#3614

With any Python version on Windows, `RUNFILES_*` environment variables still point to an intermediate stage path instead of the module path, causing `CurrentRepository()` to raise a `ValueError` since `rules_python` v1.8.0: ``` ValueError: C:\Users\bot\AppData\Local\Temp\Bazel.runfiles_gh34ij5_\runfiles\+_repo_rules+cpython\my_script.py does not lie under the runfiles root C:/cache/ab1cdef2/execroot/_main/bazel-out/x64_windows-fastbuild/bin/external/+_repo_rules+cpython/install.exe.runfiles ``` This was not the case with `rules_python` v1.7.0. The issue stems from behavior. Since bazel-contrib#3086 came up with a corresponding workaround in `//tests/runtime_env_toolchain:toolchain_runs_test`, the proposed fix simply consists in moving it to `CurrentRepository()`, thus adding another case to the workaround introduced by bazel-contrib#1634 for Python < 3.11. It therefore leads to assuming the main module path on Windows as well. Removing the workaround from `CurrentRepository()` would make the test fail as follows: ``` ==================== Test output for //tests/runtime_env_toolchain:toolchain_runs_test: E ====================================================================== ERROR: test_ran (__main__.RunTest.test_ran) ---------------------------------------------------------------------- Traceback (most recent call last): [...] ValueError: C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\tests\runtime_env_toolchain\toolchain_runs_test.py does not lie under the runfiles root c:\users\user\_bazel_user\cxxeswjo\execroot\_main\bazel-out\x64_windows-fastbuild-st-c530e4918e48\bin\tests\runtime_env_toolchain\toolchain_runs_test.exe.runfiles ``` Fixesbazel-contrib#3579. (cherry picked from commit f78add7)

With any Python version on Windows, `RUNFILES_*` environment variables still point to an intermediate stage path instead of the module path, causing `CurrentRepository()` to raise a `ValueError` since `rules_python` v1.8.0: ``` ValueError: C:\Users\bot\AppData\Local\Temp\Bazel.runfiles_gh34ij5_\runfiles\+_repo_rules+cpython\my_script.py does not lie under the runfiles root C:/cache/ab1cdef2/execroot/_main/bazel-out/x64_windows-fastbuild/bin/external/+_repo_rules+cpython/install.exe.runfiles ``` This was not the case with `rules_python` v1.7.0. The issue stems from bazel-contrib#3086 which, by eliminating a wrong assumption, also brought a stricter behavior. Since bazel-contrib#3086 came up with a corresponding workaround in `//tests/runtime_env_toolchain:toolchain_runs_test`, the proposed fix simply consists in moving it to `CurrentRepository()`, thus adding another case to the workaround introduced by bazel-contrib#1634 for Python < 3.11. It therefore leads to assuming the main module path on Windows as well. Removing the workaround from `CurrentRepository()` would make the test fail as follows: ``` ==================== Test output for //tests/runtime_env_toolchain:toolchain_runs_test: E ====================================================================== ERROR: test_ran (__main__.RunTest.test_ran) ---------------------------------------------------------------------- Traceback (most recent call last): [...] ValueError: C:\Users\user\AppData\Local\Temp\Bazel.runfiles_1f08smy6\runfiles\_main\tests\runtime_env_toolchain\toolchain_runs_test.py does not lie under the runfiles root c:\users\user\_bazel_user\cxxeswjo\execroot\_main\bazel-out\x64_windows-fastbuild-st-c530e4918e48\bin\tests\runtime_env_toolchain\toolchain_runs_test.exe.runfiles ``` Fixesbazel-contrib#3579.

This reverts commit 2c9f6f4.

jsing-canva requested review from aignas and rickeylev as code owners July 14, 2025 07:25

jsing-canva force-pushed the runfiles-fix-upstream branch from 0e21de0 to fa3209aCompare July 14, 2025 07:41

jsing-canva force-pushed the runfiles-fix-upstream branch 2 times, most recently from 45b6f72 to da225b5Compare October 23, 2025 00:59

aignas approved these changes Nov 5, 2025
View reviewed changes

jsing-canva force-pushed the runfiles-fix-upstream branch from 407ca71 to a26e18eCompare November 5, 2025 12:50

jsing-canvaand others added 4 commits November 6, 2025 14:07

Add test that ensures we can correctly locate and open a data file.
d86b9e9

Add workaround for buggy zip file bootstrap.
d632268

doc: move changelog to the unreleased section.
6166d66

jsing-canva force-pushed the runfiles-fix-upstream branch from a26e18e to 6166d66Compare November 6, 2025 03:07

Appease mypy
7e48c4e

jsing-canva force-pushed the runfiles-fix-upstream branch from e055e28 to 7e48c4eCompare November 6, 2025 03:19

aignas approved these changes Nov 6, 2025
View reviewed changes

aignas added this pull request to the merge queue Nov 6, 2025

Merged via the queue into bazel-contrib:main with commit a506d77Nov 6, 2025
4 checks passed

rdesgroppes mentioned this pull request Feb 9, 2026
fix(runfiles): assume main repository on Windows #3578
Merged

gemini-code-assistBot mentioned this pull request Feb 18, 2026
Update dependency rules_python to v1.8.4 phst/rules_elisp#1731
Merged

phst mentioned this pull request Feb 18, 2026
compile_pip_requirements on Windows (GitHub workers) fails; bug is probably in the runfiles library #3614
Open

phst added a commit to phst/rules_elisp that referenced this pull request Feb 24, 2026
Revert "Roll backbazel-contrib/rules_python#3086locally"
04ef589
This reverts commit 2c9f6f4.

acozzette mentioned this pull request Jun 29, 2026
fix(py_venv): patch bazel-runfiles root detection aspect-build/rules_py#1209
Closed

Navigation Menu

Uh oh!

Uh oh!

fix(runfiles): correct Python runfiles path assumption - #3086

fix(runfiles): correct Python runfiles path assumption#3086
aignas merged 5 commits into
bazel-contrib:mainfrom
jsing-canva:runfiles-fix-upstream

jsing-canva commented Jul 14, 2025•
edited
Loading

Uh oh!

aignas commented Oct 11, 2025

Uh oh!

jsing-canva commented Oct 23, 2025

Uh oh!

jsing-canva commented Oct 23, 2025

Uh oh!

jsing-canva commented Nov 5, 2025

Uh oh!

aignas commented Nov 5, 2025

Uh oh!

jsing-canva commented Nov 5, 2025

Uh oh!

jsing-canva commented Nov 6, 2025

Uh oh!

Uh oh!

phst commented Jan 18, 2026

Uh oh!

aignas commented Jan 19, 2026

Uh oh!

phst commented Feb 11, 2026

Uh oh!

phst commented Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

jsing-canva commented Jul 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aignas commented Oct 11, 2025

Uh oh!

jsing-canva commented Oct 23, 2025

Uh oh!

jsing-canva commented Oct 23, 2025

Uh oh!

jsing-canva commented Nov 5, 2025

Uh oh!

aignas commented Nov 5, 2025

Uh oh!

jsing-canva commented Nov 5, 2025

Uh oh!

jsing-canva commented Nov 6, 2025

Uh oh!

Uh oh!

phst commented Jan 18, 2026

Uh oh!

aignas commented Jan 19, 2026

Uh oh!

phst commented Feb 11, 2026

Uh oh!

phst commented Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jsing-canva commented Jul 14, 2025•
edited
Loading