Skip to content

ci: stop bazel truncating a failing action's output - #4524

Merged
lanluo-nvidia merged 1 commit into
pytorch:mainfrom
shoumikhin:ci-bazel-full-error-output
Aug 19, 2026
Merged

ci: stop bazel truncating a failing action's output#4524
lanluo-nvidia merged 1 commit into
pytorch:mainfrom
shoumikhin:ci-bazel-full-error-output

Conversation

@shoumikhin

@shoumikhinshoumikhin commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

The problem

When the ExecuTorch runtime wheel build fails, the log contains no compiler or
linker diagnostics at all. Bazel caps a failing action's output and prints only its
size:

stdout (.../stdout-8230) 1498425 exceeds maximum size of
--experimental_ui_max_stdouterr_bytes=1048576 bytes; skipping

So a failing build tells you that it failed and nothing about why. Diagnosing it
means guessing, or reproducing the release image locally.

The change

One line. setup.py already forwards a BAZEL_ARGS environment variable into the
bazel invocation:

command.extend(shlex.split(os.getenv("BAZEL_ARGS", "")))

so the limit can be raised without touching any build code:

export BAZEL_ARGS="--experimental_ui_max_stdouterr_bytes=33554432"

On main, that setup.py-driven build is the only bazel invocation in this workflow,
so one export covers it.

Testing

Measured on a build that failed with this applied: the log grew from 7886 to 15915
lines, and the actual linker error appeared for the first time.

[ 86%] Linking CXX shared library _portable_lib.so
ld.gold: error: .../13/libstdc++.a(functexcept.o): multiple definition of
'std::__throw_bad_array_new_length()'

That error is a separate problem and is not addressed here. The point of this change
is that it is now visible instead of hidden.

Two things worth confirming explicitly, since an experimental flag is involved:

  • The pinned bazel accepts it. .bazelversion is 8.4.2 and the run above shows no
    unrecognized option error, which matters because bazel rejects unknown flags
    outright rather than ignoring them. If a future bazel bump removes the flag, this
    line would need updating with it.
  • The value is a ceiling, not a target. 32 MiB against an observed 1.5 MiB of
    suppressed output.

What this does not show on its own

On main the build dies during CMake configure, and that failure is small, so it is
printed either way. The visibility this adds only matters once builds get far enough
to fail with large output, which is what #4523 does. The measurement above therefore
comes from a branch carrying both changes. This is still worth landing separately: it
is independent of that fix and useful for any future large failure.

The one red check, gate, is unrelated. It is flaky on main, which currently shows
three failures and one success across four runs of it at the same commit.

When the ExecuTorch runtime wheel build fails, the log contains no compiler or
linker diagnostics at all. Bazel caps a failing action's output and prints only its
size:
stdout (.../stdout-8230) 1498425 exceeds maximum size of
--experimental_ui_max_stdouterr_bytes=1048576 bytes; skipping
So a failing build reports that it failed and nothing about why.
setup.py already forwards a BAZEL_ARGS environment variable into the bazel
invocation, so the limit can be raised without touching any build code. On main
that setup.py-driven build is the only bazel invocation in this workflow, so the
one export covers it.
Measured on a build that failed after this was applied: the log grew from 7886 to
15915 lines and the actual linker error appeared for the first time. The pinned
bazel (8.4.2, from .bazelversion) accepts the option; there is no unrecognized
option error in that run, which matters because bazel rejects unknown flags
outright.

@lanluo-nvidialanluo-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@lanluo-nvidia
lanluo-nvidia merged commit 02f2667 into pytorch:mainAug 19, 2026
111 of 116 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@shoumikhin@lanluo-nvidia