Uh oh!
There was an error while loading. Please reload this page.
fix(agent-challenge): resolve TB digest manifest in installed wheel - #58
Merged
Conversation
In-process terminal_bench own_runner argv omitted --cache-root and --digest-manifest that runner.py already wires. pydantic-settings does not export CHALLENGE_* into the subprocess env, so production trials fell back to broken defaults. Mirror runner.py before the optional --model block.
…default parents[3] only worked in the src checkout; installed wheels overshot into site-packages' grandparent and lacked dataset-digest.json. force-include the golden file and resolve via importlib.resources plus an ancestor walk.
📝 WalkthroughWalkthroughThe change packages ChangesDigest manifest integration
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant terminal_bench_settings
participant own_runner_command_args
participant own_runner_backend
participant importlib_resources
terminal_bench_settings->>own_runner_command_args: provide cache and manifest settings
own_runner_command_args->>own_runner_backend: pass --cache-root and --digest-manifest
own_runner_backend->>importlib_resources: resolve agent_challenge.golden resource
importlib_resources-->>own_runner_backend: return packaged or source-checkout manifest path
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/challenges/agent-challenge/tests/test_digest_manifest_resolution.py (1)
64-110: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftExercise resolution from an installed wheel.
These tests inspect the wheel and resolve from the source checkout, but never execute
_resolve_manifest_path(None)from an installed wheel. Aparents[3]regression inside_default_digest_manifest_path()would still pass here and fail in site-packages. Build the wheel, install it into an isolated interpreter, clear the environment override, and assert the resolved path exists.Also applies to: 113-169
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/challenges/agent-challenge/tests/test_digest_manifest_resolution.py` around lines 64 - 110, The tests only validate default manifest resolution from the source checkout, so a fixed parents[3] regression could still break installed wheels. Extend the installed-wheel test coverage around _resolve_manifest_path and _default_digest_manifest_path by building the wheel, installing it into an isolated interpreter, clearing DIGEST_MANIFEST_ENV, and asserting the resolved path exists and is dataset-digest.json.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@packages/challenges/agent-challenge/tests/test_digest_manifest_resolution.py`:
- Around line 64-110: The tests only validate default manifest resolution from
the source checkout, so a fixed parents[3] regression could still break
installed wheels. Extend the installed-wheel test coverage around
_resolve_manifest_path and _default_digest_manifest_path by building the wheel,
installing it into an isolated interpreter, clearing DIGEST_MANIFEST_ENV, and
asserting the resolved path exists and is dataset-digest.json.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7995c940-7e42-45e2-8627-9dd46b16ec41
📒 Files selected for processing (5)
packages/challenges/agent-challenge/pyproject.tomlpackages/challenges/agent-challenge/src/agent_challenge/evaluation/own_runner_backend.pypackages/challenges/agent-challenge/src/agent_challenge/evaluation/terminal_bench.pypackages/challenges/agent-challenge/tests/test_digest_manifest_resolution.pypackages/challenges/agent-challenge/tests/test_evaluation.py
hatch force-include needs golden/dataset-digest.json at /app during pip install; both runtime and terminal-bench-runner stages now COPY it before install so CI image builds stop failing FileNotFoundError. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/challenges/agent-challenge/tests/test_digest_manifest_resolution.py`:
- Around line 181-212: The test around the digest manifest must validate
packaging configuration and Docker stages precisely. Parse pyproject.toml and
assert golden/dataset-digest.json is a key in Hatch’s wheel force-include
mapping, then split the Dockerfile at the runtime and terminal-bench-runner
stage declarations and require each stage to contain its own COPY
golden/dataset-digest.json before that stage’s RUN pip install --no-cache-dir .;
remove the global count and cross-stage ordering checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 0250c24f-44e1-43ff-85c1-ad2bfb97f002
📒 Files selected for processing (2)
packages/challenges/agent-challenge/Dockerfilepackages/challenges/agent-challenge/tests/test_digest_manifest_resolution.py
Comment on lines
+181
to
+212
| dockerfile = (_PKG_ROOT / "Dockerfile").read_text(encoding="utf-8") | ||
| pyproject = (_PKG_ROOT / "pyproject.toml").read_text(encoding="utf-8") | ||
| assert '"golden/dataset-digest.json"' in pyproject or ( | ||
| "golden/dataset-digest.json" in pyproject | ||
| ), "pyproject must force-include golden/dataset-digest.json" | ||
| # Exact path used by hatch force-include source key. | ||
| copy_line = "COPY golden/dataset-digest.json" | ||
| assert copy_line in dockerfile, ( | ||
| "Dockerfile must COPY golden/dataset-digest.json so hatch force-include " | ||
| "finds /app/golden/dataset-digest.json during pip install ." | ||
| ) | ||
| assert dockerfile.count(copy_line) >= 2, ( | ||
| "both runtime and terminal-bench-runner stages need the golden COPY " | ||
| f"(found {dockerfile.count(copy_line)})" | ||
| ) | ||
| # Each pip install of the package must be preceded by the golden COPY in-file order. | ||
| install_marker = "RUN pip install --no-cache-dir ." | ||
| positions = [] | ||
| start = 0 | ||
| while True: | ||
| idx = dockerfile.find(install_marker, start) | ||
| if idx < 0: | ||
| break | ||
| positions.append(idx) | ||
| start = idx + len(install_marker) | ||
| assert positions, "expected at least one package pip install in Dockerfile" | ||
| for idx in positions: | ||
| preceding = dockerfile[:idx] | ||
| assert copy_line in preceding, ( | ||
| "COPY golden/dataset-digest.json must appear before each " | ||
| "RUN pip install --no-cache-dir ." | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the packaging regression assertions stage-aware.
Both COPY lines could be placed in runtime while terminal-bench-runner has none: the global count and “any preceding COPY” checks would still pass. Also parse pyproject.toml and assert the entry is in Hatch’s wheel force-include mapping, rather than accepting the path anywhere in the file. Split the Dockerfile into the two named stages and require the COPY before pip install . within each stage.
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] 200-200: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: dockerfile.find(install_marker, start)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/challenges/agent-challenge/tests/test_digest_manifest_resolution.py`
around lines 181 - 212, The test around the digest manifest must validate
packaging configuration and Docker stages precisely. Parse pyproject.toml and
assert golden/dataset-digest.json is a key in Hatch’s wheel force-include
mapping, then split the Dockerfile at the runtime and terminal-bench-runner
stage declarations and require each stage to contain its own COPY
golden/dataset-digest.json before that stage’s RUN pip install --no-cache-dir .;
remove the global count and cross-stage ordering checks.
Uh oh!
There was an error while loading. Please reload this page.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
failure_stage: jobwithTaskDefNotFoundError: digest manifest missing: /usr/local/lib/python3.12/golden/dataset-digest.json(reason_code=terminal_bench_failed).Symptom
Production path never reached agent load; score was always 0/30.
Root cause (three defects)
terminal_bench.own_runner_command_argsdid not pass--cache-root/--digest-manifest, while the equivalent builder inrunner.py(~L1964–1967) already did. The production TB attempt path uses the former._resolve_manifest_pathfell back toPath(__file__).resolve().parents[3] / "golden" / "dataset-digest.json", correct only for thesrc/checkout layout; in an installed wheel it overshoots intosite-packages' grandparent (/usr/local/lib/python3.12/golden/...).golden/dataset-digest.jsonwas never shipped as wheel package data.Why flags matter: pydantic-settings reads
embed.envinto the settings object but does not export toos.environ, so the spawned runner subprocess never inherited the correct paths. Passing them explicitly as CLI flags is the durable bridge.Changes
own_runner_command_args: forward--cache-rootand--digest-manifest(parity withrunner.py)._resolve_manifest_path: layout-safe default that works for both checkout and installed wheel.pyproject.toml: shipgolden/dataset-digest.jsonas package data so the wheel containsagent_challenge/golden/dataset-digest.json.Test plan
own_runner or terminal_benchselection — 672 passed / 3 skippedagent_challenge/golden/dataset-digest.jsonProd mitigation note
A temporary symlink mitigation is currently live in production and will be removed after this deploys. This PR is the durable fix; do not leave the symlink as the long-term path.
CI
Notes
6bfd68af,98e55aebcompose.envchangesSummary by CodeRabbit
Bug Fixes
Reliability
Tests