Uh oh!
There was an error while loading. Please reload this page.
fix(harness): survive binary tool output and slow container teardown - #6
Open
JeremyJC67 wants to merge 2 commits into
Open
fix(harness): survive binary tool output and slow container teardown#6JeremyJC67 wants to merge 2 commits into
JeremyJC67 wants to merge 2 commits into
Conversation
added 2 commits
September 2, 2026 21:57
_exec_in_container decoded docker exec output with strict UTF-8, so any command emitting raw bytes (xxd -r -p, openssl, binary tokens) raised UnicodeDecodeError and aborted the whole task run. The task was then recorded as an error rather than scored, silently dropping it from the denominator. Decode with errors=replace so the agent sees the output and the task is scored normally.
_stop_container ran docker stop with a 30s timeout and let TimeoutExpired propagate. On a loaded host teardown can exceed that, and the uncaught exception killed an entire multi-run experiment after the first task. Raise the timeout, fall back to docker rm -f, and swallow teardown failures - cleanup must never decide whether the experiment completes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two robustness fixes to the evaluation harness, both surfaced while running a full 3-run gpt-4o sweep of all 33 tasks.
Neither changes task content, verifiers, or scoring logic. Both turn a crash into a normal result, so they can only move numbers by letting tasks be scored that previously were not.
1. Strict UTF-8 decoding aborted tasks on binary tool output
_exec_in_containerransubprocess.run(..., text=True)with noerrors=argument. Any command the agent issues that emits raw bytes —xxd -r -p,opensslraw output, binary tokens — raisedUnicodeDecodeError, which propagated and killed the whole task run.The task was then recorded as an error rather than scored, so it silently dropped out of the denominator instead of counting as a failure. Observed on
oauth-authorization-code:Fixed by decoding with
errors="replace", so the agent sees the output and the task is scored normally.2. An uncaught teardown timeout aborted entire multi-run experiments
_stop_containerrandocker stopwith a 30-second timeout and letTimeoutExpiredpropagate. On a loaded host teardown can exceed that, and the uncaught exception killed a whole--k 3experiment after its first task:Fixed by raising the timeout, falling back to
docker rm -f, and swallowing teardown failures. Cleanup should never decide whether an experiment completes.Verification
Before: a
--k 3sweep aborted after 1 task; a single-run sweep reportedSummary: 11/33 passed, 1 errors.After:
Summary: 32/99 passed, 0 errorsacross the full 3-run sweep — 99 trials, zero harness errors.