🚀 feature request
Relevant Rules
py_binary, py_test (from @rules_python)
Description
When building Python binaries or tests using Bazel (specifically with py_binary and py_test),
we encounter an "Argument list too long" error during the build process.
This happens when our projects depend on a very large number of files, particularly those from large Python libraries managed by pip_parse (e.g., boto3, msgraph-sdk-python).
The root cause seems to be that Bazel calls the zipper tool by passing all file paths to be included in the package directly as command-line arguments.
| def_create_zip_file(ctx, *, output, original_nonzip_executable, zip_main, runfiles): |
| """Create a Python zipapp (zip with __main__.py entry point).""" |
| workspace_name=ctx.workspace_name |
| legacy_external_runfiles=_py_builtins.get_legacy_external_runfiles(ctx) |
| |
| manifest=ctx.actions.args() |
| manifest.use_param_file("@%s", use_always=True) |
| manifest.set_param_file_format("multiline") |
| |
| manifest.add("__main__.py={}".format(zip_main.path)) |
| manifest.add("__init__.py=") |
| manifest.add( |
| "{}=".format( |
| _get_zip_runfiles_path("__init__.py", workspace_name, legacy_external_runfiles), |
| ), |
| ) |
| forpathinrunfiles.empty_filenames.to_list(): |
| manifest.add("{}=".format(_get_zip_runfiles_path(path, workspace_name, legacy_external_runfiles))) |
| |
| defmap_zip_runfiles(file): |
| iffile!=original_nonzip_executableandfile!=output: |
| return"{}={}".format( |
| _get_zip_runfiles_path(file.short_path, workspace_name, legacy_external_runfiles), |
| file.path, |
| ) |
| else: |
| returnNone |
| |
| manifest.add_all(runfiles.files, map_each=map_zip_runfiles, allow_closure=True) |
| |
| inputs= [zip_main] |
| if_py_builtins.is_bzlmod_enabled(ctx): |
| zip_repo_mapping_manifest=ctx.actions.declare_file( |
| output.basename+".repo_mapping", |
| sibling=output, |
| ) |
| _py_builtins.create_repo_mapping_manifest( |
| ctx=ctx, |
| runfiles=runfiles, |
| output=zip_repo_mapping_manifest, |
| ) |
| manifest.add("{}/_repo_mapping={}".format( |
| _ZIP_RUNFILES_DIRECTORY_NAME, |
| zip_repo_mapping_manifest.path, |
| )) |
| inputs.append(zip_repo_mapping_manifest) |
| |
| forartifactinrunfiles.files.to_list(): |
| # Don't include the original executable because it isn't used by the |
| # zip file, so no need to build it for the action. |
| # Don't include the zipfile itself because it's an output. |
| ifartifact!=original_nonzip_executableandartifact!=output: |
| inputs.append(artifact) |
| |
| zip_cli_args=ctx.actions.args() |
| zip_cli_args.add("cC") |
| zip_cli_args.add(output) |
| |
| ctx.actions.run( |
| executable=ctx.executable._zipper, |
| arguments= [zip_cli_args, manifest], |
| inputs=depset(inputs), |
| outputs= [output], |
| use_default_shell_env=True, |
| mnemonic="PythonZipper", |
| progress_message="Building Python zip: %{label}", |
| ) |
This leads to the argument list exceeding the operating system's ARG_MAX limit.
This error makes our Bazel builds unstable and undermines the reliability of our CI/CD pipelines for large Python projects.
Describe the solution you'd like
We propose enhancing rules_python to always pass arguments to the zipper tool via a temporary response file, rather than directly on the command line.
Upon inspecting zipper's zip_main.cc source code, it appears to support reading arguments from a file using the @ syntax (as indicated by logic to process arguments starting with @).
By consistently utilizing this response file capability, rules_python can entirely bypass OS ARG_MAX limitations.
🚀 feature request
Relevant Rules
py_binary,py_test(from @rules_python)Description
When building Python binaries or tests using Bazel (specifically with py_binary and py_test),
we encounter an "Argument list too long" error during the build process.
This happens when our projects depend on a very large number of files, particularly those from large Python libraries managed by pip_parse (e.g., boto3, msgraph-sdk-python).
The root cause seems to be that Bazel calls the zipper tool by passing all file paths to be included in the package directly as command-line arguments.
rules_python/python/private/py_executable.bzl
Lines 891 to 957 in 9429ae6
This leads to the argument list exceeding the operating system's ARG_MAX limit.
This error makes our Bazel builds unstable and undermines the reliability of our CI/CD pipelines for large Python projects.
Describe the solution you'd like
We propose enhancing rules_python to always pass arguments to the zipper tool via a temporary response file, rather than directly on the command line.
Upon inspecting zipper's zip_main.cc source code, it appears to support reading arguments from a file using the @ syntax (as indicated by logic to process arguments starting with @).
By consistently utilizing this response file capability, rules_python can entirely bypass OS ARG_MAX limitations.