Uh oh!
There was an error while loading. Please reload this page.
fix(prism): stop false stuck-sweep and retain harness log tails - #88
Merged
Conversation
A healthy ~7h19m train was killed at the old 7h grace (updated_at never heartbeats during SSH), with no on-pod log harvest and a 4 KiB head truncate that ate the real fatal. Raise grace to 10h, tee+harvest logs before reclaim, and keep a 32 KiB stderr/log tail.
📝 WalkthroughWalkthroughThe change adds bounded harness-log retention for live evaluations, updates SSH diagnostics, and integrates log harvesting into stuck-job recovery. The stuck-job grace period increases from 7 to 10 hours. ChangesHarness retention and stuck-job recovery
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant StuckSweeper
participant EvalJobBackend
participant Pod
StuckSweeper->>EvalJobBackend: Harvest logs for stuck submission
EvalJobBackend->>Pod: Fetch retained harness log tail
Pod-->>EvalJobBackend: Return bounded log tail
EvalJobBackend-->>StuckSweeper: Add logs to failure details
StuckSweeper->>StuckSweeper: Auto-retry infrastructure failure or mark terminal failure
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/prism-challenge/src/orchestrator.rs`:
- Around line 211-229: The stuck-pod cleanup flow must not call maybe_auto_retry
or fail_terminal until termination is confirmed. In the block handling
row.pod_id, capture terminate and verify_terminated results, and only proceed to
retry or terminal handling when termination succeeds and verification confirms
the pod is absent; otherwise leave the row non-retryable so a later sweep can
retry cleanup.
In `@crates/prism-lium/src/client.rs`:
- Around line 500-503: Extend the harness failure coverage around the
error-detail construction in the client execution flow to persist a full
HARNESS_LOG_RETAIN_BYTES tail, including a negative/schema test that exercises
the expanded prism_submission.error_detail and prism_stage_event.detail fields.
Validate that the API response and stage-event JSON serialize the complete
error_detail without truncation or schema mismatch.
In `@docs/PRISM.md`:
- Around line 62-66: Update the sweeper behavior description in the PRISM
documentation to state that it first invokes maybe_auto_retry for eligible stuck
rows, returning them to Queued while retry budget remains. Specify that rows
become terminal ChallengeInternal failures only after the automatic-retry budget
is exhausted, while preserving the existing log-harvesting and recover_on_boot
details.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 6e89ac1e-3cb6-4fe6-b748-ef1c60e71bc0
📒 Files selected for processing (7)
bins/prism-challenge/src/main.rscrates/prism-challenge/src/orchestrator.rscrates/prism-lium/src/client.rscrates/prism-lium/src/lib.rscrates/prism-lium/src/ssh.rsdocs/COMPLETENESS.mddocs/PRISM.md
Comment on lines
211
to
+229
| if let Some(pod) = row.pod_id.clone() { | ||
| let _ = self.backend.terminate(&pod).await; | ||
| let _ = self.backend.verify_terminated(&pod).await; | ||
| } | ||
| let id = row.id.clone(); | ||
| let _ = self | ||
| .store | ||
| .apply( | ||
| &id, | ||
| &StatePatch { | ||
| status: Some(Stage::Failed), | ||
| error_detail: Some("swept: stuck beyond grace".into()), | ||
| retry_bump: 1, | ||
| ..StatePatch::default() | ||
| }, | ||
| Some(&StageEvent { | ||
| stage: Stage::Failed, | ||
| detail: Some(serde_json::json!({"reason": "stuck-sweep"})), | ||
| at_ms: 0, | ||
| }), | ||
| let msg = if harvested.trim().is_empty() { | ||
| "swept: stuck beyond grace".into() | ||
| } else { | ||
| format!( | ||
| "swept: stuck beyond grace; harvested: {}", | ||
| prism_lium::truncate_tail(&harvested, prism_lium::HARNESS_LOG_RETAIN_BYTES) | ||
| ) | ||
| .await; | ||
| }; | ||
| // Infra-class: auto-retry while budget remains (do **not** burn a | ||
| // retry_bump without requeue — that previously exhausted manual | ||
| // retry while leaving gating `registered`). | ||
| if self.maybe_auto_retry(&row, "install", &msg).await { | ||
| continue; | ||
| } | ||
| self.fail_terminal(&row, "install", &msg).await; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Do not requeue until pod termination is verified.
If terminate fails or verify_terminated returns false, this code still calls maybe_auto_retry. A new worker can then provision another pod while the stuck pod continues training. This can duplicate evaluation work and provider billing.
Keep the row non-retryable until termination succeeds and verification confirms that the pod is absent. Let a later sweep retry cleanup.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/prism-challenge/src/orchestrator.rs` around lines 211 - 229, The
stuck-pod cleanup flow must not call maybe_auto_retry or fail_terminal until
termination is confirmed. In the block handling row.pod_id, capture terminate
and verify_terminated results, and only proceed to retry or terminal handling
when termination succeeds and verification confirms the pod is absent; otherwise
leave the row non-retryable so a later sweep can retry cleanup.
Comment on lines
500
to
+503
| return Err(LiumError::Exec(format!( | ||
| "harness failed (code {}): {}", | ||
| out.returncode, | ||
| truncate(&out.stderr, 4000) | ||
| truncate_tail(&detail, HARNESS_LOG_RETAIN_BYTES) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
# Map the store implementation before inspecting its persistence contracts.
ast-grep outline crates/prism-challenge/src --items all
# Inspect all Rust and SQL references to the affected persisted fields.
rg -n -C 5 '\berror_detail\b|prism_submission|prism_stage_event|StatePatch' \
crates bins -g '*.rs' -g '*.sql'# Locate migrations that can constrain the new retained-log size.
fd -t f -e sql .| xargs -r rg -n -C 3 \
'error_detail|prism_submission|prism_stage_event'Repository: BaseIntelligence/base
Length of output: 50378
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"## candidate files"
fd -t f '^(client\.rs|lib\.rs|dbprism\.rs|store\.rs|dbs\.rs|api\.rs)$' crates/prism-lium crates/prism-store crates/prism-challenge bins/prism-challenge
echoecho"## prism-lium harness tail and fail path"
ast-grep outline crates/prism-lium/src/client.rs --match fail_terminal --view expanded ||true
rg -n -C 8 'HARNESS_LOG_RETAIL|failure|err|fail_terminal|returncode' crates/prism-lium/src/client.rs crates/prism-lium/src/lib.rs
echoecho"## prism-store db schema/persistence references"
fd -t f -e sql .| xargs -r rg -n -C 4 'CREATE TABLE.*prism_(submission|stage_event)|prism_(submission|stage_event).*CREATE|error_detail|detail'
sed -n '/PrismSubmissionRow/,/impl PrismSubmissionRow/p' crates/prism-store/src/db.rs 2>/dev/null ||true
sed -n '1,260p' crates/prism-store/src/dbprism.rs
echoecho"## prism-challenge API persistence contract"
ast-grep outline crates/prism-challenge/src/api.rs --view expanded ||true
sed -n '/post_retry/,/}/p' crates/prism-challenge/src/api.rs
sed -n '/pub fn get_status/,/pub fn get_events/p' crates/prism-challenge/src/api.rs
rg -n -C 5 'post_retry|get_status|get_events|fail_terminal|apply\(' crates/prism-challenge/src -g '*.rs'Repository: BaseIntelligence/base
Length of output: 50377
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"## prism-store db.rs relevant generated/update functions"
sed -n '1,280p' crates/db/src/prism_store.rs
sed -n '280,520p' crates/db/src/prism_store.rs
echoecho"## prism-lium fail_terminal call sites"
rg -n -C 8 'fail_terminal|Err\(LiumError::Exec|LiumError::Exec\(' crates/prism-challenge src crates/prism-lium crates/prism -g '*.rs'echoecho"## prism-store trait/struct definitions"
sed -n '145,210p' crates/prism-store/src/store.rs
sed -n '440,505p' crates/prism-store/src/store.rs
sed -n '980,1040p' crates/prism-store/src/store.rs
echoecho"## relevant API implementation slices"
sed -n '370,400p' crates/prism-challenge/src/api.rs
sed -n '402,432p' crates/prism-challenge/src/api.rsRepository: BaseIntelligence/base
Length of output: 27622
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"## orchestrator failure path around LiumClient.measure/eval"
sed -n '216,250p' crates/prism-challenge/src/orchestrator.rs
sed -n '340,380p' crates/prism-challenge/src/orchestrator.rs
sed -n '304,348p' crates/prism-challenge/src/orchestrator.rs
echoecho"## StageEvent and memory store apply implementation"
sed -n '152,166p' crates/prism-store/src/store.rs
sed -n '444,492p' crates/prism-store/src/store.rs
echoecho"## fail_terminal implementations/call chain in orchestrator"
rg -n -C 10 'maybe_auto_retry|fail_terminal|apply\(' crates/prism-challenge/src/orchestrator.rs crates/prism-challenge/src/api.rs crates/prism-challenge/src/lib.rs -g '*.rs'Repository: BaseIntelligence/base
Length of output: 34221
Expand the 32 KiB harness diagnostic contract into persistence coverage.
prism_submission.error_detail and prism_stage_event.detail can now store a full HARNESS_LOG_RETAIN_BYTES tail. Add a negative/schema test for this path and validate API/event serialization around the expanded error_detail and stage JSON.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/prism-lium/src/client.rs` around lines 500 - 503, Extend the harness
failure coverage around the error-detail construction in the client execution
flow to persist a full HARNESS_LOG_RETAIN_BYTES tail, including a
negative/schema test that exercises the expanded prism_submission.error_detail
and prism_stage_event.detail fields. Validate that the API response and
stage-event JSON serialize the complete error_detail without truncation or
schema mismatch.
Comment on lines
+62
to
+66
| lives in `prism_submission`. The sweeper fails rows stuck past the **10h** | ||
| grace (aligned above wait-RUNNING + 6h train + SSH margin; a prior 7h grace | ||
| false-positive swept healthy ~7h19m trains) as `ChallengeInternal` after | ||
| harvesting the on-pod harness log tail, and `recover_on_boot` cleans pods | ||
| referenced by interrupted rows. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Document the automatic retry branch.
The sweeper does not immediately fail every stuck row as ChallengeInternal. It first calls maybe_auto_retry, which returns eligible rows to Queued. State that terminal ChallengeInternal failure occurs only after the auto-retry budget is exhausted.
As per coding guidelines, “Treat normative documentation—including architecture files, frozen specifications, threat and operator-security documents, completeness status, runbooks, and external-miner/—as the source of truth for contracts, operations, and status.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/PRISM.md` around lines 62 - 66, Update the sweeper behavior description
in the PRISM documentation to state that it first invokes maybe_auto_retry for
eligible stuck rows, returning them to Queued while retry budget remains.
Specify that rows become terminal ChallengeInternal failures only after the
automatic-retry budget is exhausted, while preserving the existing
log-harvesting and recover_on_boot details.
Source: Coding guidelines
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
9f77f258…killed at 7h19m with blank attempt-2 logs)./tmp/prism_eval/harness.log, harvest the tail before reclaim/timeout, and keep a 32 KiB tail of stderr/logs in error_detail (was a 4 KiB head truncate that ate inductor spam and lost the fatal).fail_terminal) instead of burningretry_bumpwithout requeue.Test plan
cargo test -p prism-lium --libcargo test -p prism-challenge --libcargo clippy -p prism-lium -p prism-challenge -p prism-challenge-bin --all-targets -- -D warningscargo run -p xtask -- spec-check/design-checkprism-challengeto prod; confirm grace=10h in running binary / no false sweeps on long trains12af6c7c…(uid 149) — slot stillregistered,retry_count=2blocks manual retrySummary by CodeRabbit
New Features
Bug Fixes
Documentation