fix: don't require system Python to perform bootstrapping

CHANGELOG.md

@@ -1,3 +1,6 @@
+:::{default-domain} bzl
+:::
+
 # rules_python Changelog
 
 This is a human-friendly changelog in a keepachangelog.com style format.
@@ -31,7 +34,7 @@ A brief description of the categories of changes:
  marked as `reproducible` and will not include any lock file entries from now
  on.
 
-* (gazelle): Remove gazelle plugin's python deps and make it hermetic.
+* (gazelle): Remove gazelle plugin's python deps and make it hermetic.
  Introduced a new Go-based helper leveraging tree-sitter for syntax analysis.
  Implemented the use of `pypi/stdlib-list` for standard library module verification.
 
@@ -80,6 +83,16 @@ A brief description of the categories of changes:
  invalid usage previously but we were not failing the build. From now on this
  is explicitly disallowed.
 * (toolchains) Added riscv64 platform definition for python toolchains.
+* (rules) A new bootstrap implementation that doesn't require a system Python
+ is available. It can be enabled by setting
+ {obj}`--@rules_python//python:config_settings:bootstrap_impl=two_phase`. It
+ will become the default in a subsequent release.
+ ([#691](https://github.com/bazelbuild/rules_python/issues/691))
+* (providers) `PyRuntimeInfo` has two new attributes:
+ {obj}`PyRuntimeInfo.stage2_bootstrap_template` and
+ {obj}`PyRuntimeInfo.zip_main_template`.
+* (toolchains) A replacement for the Bazel-builtn autodetecting toolchain is
+ available. The `//python:autodetecting_toolchain` alias now uses it.
 
 [precompile-docs]: /precompiling
 

CONTRIBUTING.md

@@ -175,6 +175,7 @@ Issues should be triaged as follows:
  functionality, should also be filed in this repository but without the
  `core-rules` label.
 
+(breaking-changes)=
 ## Breaking Changes
 
 Breaking changes are generally permitted, but we follow a 3-step process for

docs/sphinx/api/python/config_settings/index.md

@@ -1,3 +1,5 @@
+:::{default-domain} bzl
+:::
 :::{bzl:currentfile} //python/config_settings:BUILD.bazel
 :::
 
@@ -66,3 +68,32 @@ Values:
 * `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
 * `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
 :::
+
+::::{bzl:flag} bootstrap_impl
+Determine how programs implement their startup process.
+
+Values:
+* `system_python`: Use a bootstrap that requires a system Python available
+ in order to start programs. This requires
+ {obj}`PyRuntimeInfo.bootstrap_template` to be a Python program.
+* `script`: Use a bootstrap that uses an arbitrary executable script (usually a
+ shell script) instead of requiring it be a Python program.
+
+:::{note}
+The `script` bootstrap requires the toolchain to provide the `PyRuntimeInfo`
+provider from `rules_python`. This loosely translates to using Bazel 7+ with a
+toolchain created by rules_python. Most notably, WORKSPACE builds default to
+using a legacy toolchain built into Bazel itself which doesn't support the
+script bootstrap. If not available, the `system_python` bootstrap will be used
+instead.
+:::
+
+:::{seealso}
+{obj}`PyRuntimeInfo.bootstrap_template` and
+{obj}`PyRuntimeInfo.stage2_bootstrap_template`
+:::
+
+:::{versionadded} 0.33.0
+:::
+
+::::

docs/sphinx/api/python/index.md

@@ -1,3 +1,5 @@
+:::{default-domain} bzl
+:::
 :::{bzl:currentfile} //python:BUILD.bazel
 :::
 
@@ -21,3 +23,21 @@ provides:
 * `PyRuntimeInfo`: The consuming target's target toolchain information
 
 :::
+
+::::{target} autodetecting_toolchain
+
+A simple toolchain that simply uses `python3` from the runtime environment.
+
+Note that this toolchain provides no build-time information, which makes it of
+limited utility.
+
+This is only provided to aid migration off the builtin Bazel toolchain 
+(`@bazel_tools//python:autodetecting_toolchain`), and is largely only applicable
+to WORKSPACE builds.
+
+:::{deprecated} unspecified
+
+Switch to using a hermetic toolchain or manual toolchain configuration instead.
+:::
+
+::::

docs/sphinx/bazel_inventory.txt

@@ -10,7 +10,7 @@ bool bzl:type 1 rules/lib/bool -
 int bzl:type 1 rules/lib/int -
 depset bzl:type 1 rules/lib/depset -
 dict bzl:type 1 rules/lib/dict -
-label bzl:doc 1 concepts/labels -
+label bzl:type 1 concepts/labels -
 attr.bool bzl:type 1 rules/lib/toplevel/attr#bool -
 attr.int bzl:type 1 rules/lib/toplevel/attr#int -
 attr.label bzl:type 1 rules/lib/toplevel/attr#label -
@@ -21,6 +21,7 @@ list bzl:type 1 rules/lib/list -
 python bzl:doc 1 reference/be/python -
 str bzl:type 1 rules/lib/string -
 struct bzl:type 1 rules/lib/builtins/struct -
-target-name bzl:doc 1 concepts/labels#target-names -
+Name bzl:type 1 concepts/labels#target-names -
 CcInfo bzl:provider 1 rules/lib/providers/CcInfo -
 CcInfo.linking_context bzl:provider-field 1 rules/lib/providers/CcInfo#linking_context -
+ToolchainInfo bzl:type 1 rules/lib/providers/ToolchainInfo.html -

docs/sphinx/pip.md

@@ -150,7 +150,7 @@ ARG=$1 # but we don't do anything with it as it's always "get"
 # formatting is optional
 echo '{'
 echo ' "headers": {'
-echo ' "Authorization": ["Basic dGVzdDoxMjPCow=="]
+echo ' "Authorization": ["Basic dGVzdDoxMjPCow=="]'
 echo ' }'
 echo '}'
 ```

docs/sphinx/support.md

@@ -46,7 +46,8 @@ incremental fashion.
 
 Breaking changes are allowed, but follow a process to introduce them over
 a series of releases to so users can still incrementally upgrade. See the
-[Breaking Changes](contributing#breaking-changes) doc for the process.
+[Breaking Changes](#breaking-changes) doc for the process.
+
 
 ## Experimental Features
 

docs/sphinx/toolchains.md

@@ -1,3 +1,6 @@
+:::{default-domain} bzl
+:::
+
 # Configuring Python toolchains and runtimes
 
 This documents how to configure the Python toolchain and runtimes for different
@@ -193,7 +196,7 @@ load("@rules_python//python:repositories.bzl", "py_repositories")
 py_repositories()
 ```
 
-#### Workspace toolchain registration
+### Workspace toolchain registration
 
 To register a hermetic Python toolchain rather than rely on a system-installed interpreter for runtime execution, you can add to the `WORKSPACE` file:
 
@@ -221,3 +224,21 @@ pip_parse(
 After registration, your Python targets will use the toolchain's interpreter during execution, but a system-installed interpreter
 is still used to 'bootstrap' Python targets (see https://github.com/bazelbuild/rules_python/issues/691).
 You may also find some quirks while using this toolchain. Please refer to [python-build-standalone documentation's _Quirks_ section](https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html).
+
+## Autodetecting toolchain
+
+The autodetecting toolchain is a deprecated toolchain that is built into Bazel.
+It's name is a bit misleading: it doesn't autodetect anything. All it does is
+use `python3` from the environment a binary runs within. This provides extremely
+limited functionality to the rules (at build time, nothing is knowable about
+the Python runtime).
+
+Bazel itself automatically registers `@bazel_tools//python:autodetecting_toolchain`
+as the lowest priority toolchain. For WORKSPACE builds, if no other toolchain
+is registered, that toolchain will be used. For bzlmod builds, rules_python
+automatically registers a higher-priority toolchain; it won't be used unless
+there is a toolchain misconfiguration somewhere.
+
+To aid migration off the Bazel-builtin toolchain, rules_python provides
+{obj}`@rules_python//python:autodetecting_toolchain`. This is an equivalent
+toolchain, but is implemented using rules_python's objects.

examples/bzlmod/test.py

@@ -14,6 +14,7 @@
 
 import os
 import pathlib
+import re
 import sys
 import unittest
 
@@ -63,16 +64,47 @@ def test_coverage_sys_path(self):
  first_item.endswith("coverage"),
  f"Expected the first item in sys.path '{first_item}' to not be related to coverage",
  )
+
+ # We're trying to make sure that the coverage library added by the
+ # toolchain is _after_ any user-provided dependencies. This lets users
+ # override what coverage version they're using.
+ first_coverage_index = None
+ last_user_dep_index = None
+ for i, path in enumerate(sys.path):
+ if re.search("rules_python.*~pip~", path):
+ last_user_dep_index = i
+ if first_coverage_index is None and re.search(
+ ".*rules_python.*~python~.*coverage.*", path
+ ):
+ first_coverage_index = i
+
  if os.environ.get("COVERAGE_MANIFEST"):
+ self.assertIsNotNone(
+ first_coverage_index,
+ "Expected to find toolchain coverage, but "
+ + f"it was not found.\nsys.path:\n{all_paths}",
+ )
+ self.assertIsNotNone(
+ first_coverage_index,
+ "Expected to find at least one uiser dep, "
+ + "but none were found.\nsys.path:\n{all_paths}",
+ )
  # we are running under the 'bazel coverage :test'
- self.assertTrue(
- "_coverage" in last_item,
- f"Expected {last_item} to be related to coverage",
+ self.assertGreater(
+ first_coverage_index,
+ last_user_dep_index,
+ "Expected coverage provided by the toolchain to be after "
+ + "user provided dependencies.\n"
+ + f"Found coverage at index: {first_coverage_index}\n"
+ + f"Last user dep at index: {last_user_dep_index}\n"
+ + f"Full sys.path:\n{all_paths}",
  )
- self.assertEqual(pathlib.Path(last_item).name, "coverage")
  else:
- self.assertFalse(
- "coverage" in last_item, f"Expected coverage tooling to not be present"
+ self.assertIsNone(
+ first_coverage_index,
+ "Expected toolchain coverage to not be present\n"
+ + f"Found coverage at index: {first_coverage_index}\n"
+ + f"Full sys.path:\n{all_paths}",
  )
 
  def test_main(self):

python/BUILD.bazel

@@ -24,6 +24,7 @@ that @rules_python//python is only concerned with the core rules.
 """
 
 load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+load("//python/private:autodetecting_toolchain.bzl", "define_autodetecting_toolchain")
 load(":current_py_toolchain.bzl", "current_py_toolchain")
 
 package(default_visibility = ["//visibility:public"])
@@ -318,14 +319,11 @@ toolchain_type(
 # safe if you know for a fact that your build is completely compatible with the
 # version of the `python` command installed on the target platform.
 
-alias(
- name = "autodetecting_toolchain",
- actual = "@bazel_tools//tools/python:autodetecting_toolchain",
-)
+define_autodetecting_toolchain(name = "autodetecting_toolchain")
 
 alias(
  name = "autodetecting_toolchain_nonstrict",
- actual = "@bazel_tools//tools/python:autodetecting_toolchain_nonstrict",
+ actual = ":autodetecting_toolchain",
 )
 
 # ========= Packaging rules =========

python/config_settings/BUILD.bazel

@@ -1,6 +1,7 @@
 load("@bazel_skylib//rules:common_settings.bzl", "string_flag")
 load(
  "//python/private:flags.bzl",
+ "BootstrapImplFlag",
  "PrecompileAddToRunfilesFlag",
  "PrecompileFlag",
  "PrecompileSourceRetentionFlag",
@@ -52,3 +53,11 @@ string_flag(
  # NOTE: Only public because its an implicit dependency
  visibility = ["//visibility:public"],
 )
+
+string_flag(
+ name = "bootstrap_impl",
+ build_setting_default = BootstrapImplFlag.SYSTEM_PYTHON,
+ values = sorted(BootstrapImplFlag.__members__.values()),
+ # NOTE: Only public because its an implicit dependency
+ visibility = ["//visibility:public"],
+)

python/private/BUILD.bazel

@@ -376,9 +376,54 @@ exports_files(
  visibility = ["//visibility:public"],
 )
 
+filegroup(
+ name = "stage1_bootstrap_template",
+ srcs = ["stage1_bootstrap_template.sh"],
+ # Not actually public. Only public because it's an implicit dependency of
+ # py_runtime.
+ visibility = ["//visibility:public"],
+)
+
+filegroup(
+ name = "stage2_bootstrap_template",
+ srcs = ["stage2_bootstrap_template.py"],
+ # Not actually public. Only public because it's an implicit dependency of
+ # py_runtime.
+ visibility = ["//visibility:public"],
+)
+
+filegroup(
+ name = "zip_main_template",
+ srcs = ["zip_main_template.py"],
+ # Not actually public. Only public because it's an implicit dependency of
+ # py_runtime.
+ visibility = ["//visibility:public"],
+)
+
+# NOTE: Windows builds don't use this bootstrap. Instead, a native Windows
+# program locates some Python exe and runs `python.exe foo.zip` which
+# runs the __main__.py in the zip file.
+alias(
+ name = "bootstrap_template",
+ actual = select({
+ ":is_script_bootstrap_enabled": "stage1_bootstrap_template.sh",
+ "//conditions:default": "python_bootstrap_template.txt",
+ }),
+ # Not actually public. Only public because it's an implicit dependency of
+ # py_runtime.
+ visibility = ["//visibility:public"],
+)
+
 # Used to determine the use of `--stamp` in Starlark rules
 stamp_build_setting(name = "stamp")
 
+config_setting(
+ name = "is_script_bootstrap_enabled",
+ flag_values = {
+ "//python/config_settings:bootstrap_impl": "script",
+ },
+)
+
 print_toolchains_checksums(name = "print_toolchains_checksums")
 
 # Used for py_console_script_gen rule

python/private/autodetecting_toolchain.bzl

@@ -32,7 +32,7 @@ def define_autodetecting_toolchain(name):
  # buildifier: disable=native-py
  py_runtime(
  name = "_autodetecting_py3_runtime",
- interpreter = ":py3wrapper.sh",
+ interpreter = "//python/private:autodetecting_toolchain_interpreter.sh",
  python_version = "PY3",
  stub_shebang = "#!/usr/bin/env python3",
  visibility = ["//visibility:private"],

python/private/common/common.bzl

@@ -182,7 +182,7 @@ def create_cc_details_struct(
  cc_toolchain = cc_toolchain,
  )
 
-def create_executable_result_struct(*, extra_files_to_build, output_groups):
+def create_executable_result_struct(*, extra_files_to_build, output_groups, extra_runfiles = None):
  """Creates a `CreateExecutableResult` struct.
 
  This is the return value type of the semantics create_executable function.
@@ -192,13 +192,15 @@ def create_executable_result_struct(*, extra_files_to_build, output_groups):
  included as default outputs.
  output_groups: dict[str, depset[File]]; additional output groups that
  should be returned.
+ extra_runfiles: A runfiles object of additional runfiles to include.
 
  Returns:
  A `CreateExecutableResult` struct.
  """
  return struct(
  extra_files_to_build = extra_files_to_build,
  output_groups = output_groups,
+ extra_runfiles = extra_runfiles,
  )
 
 def union_attrs(*attr_dicts, allow_none = False):