fix(collector): retry uploader readback on a 600s deadline, not 12 attempts - #575
Conversation
…tempts Fourth identical production failure (2026-08-01T18:23 CST): a 109MiB multipart shadow upload took ~150s to become HEAD-able, exhausting the 12x5s loop at ~90s. Every observed readback failure has been inside the uploader's own post-PUT verification (the .upload-staging/.oss-verify path), never the gate script's. Retry up to 60 attempts bounded by a 600s wall-clock budget with budget-clamped sleeps; per-file timeout stays 60s. Refs #556
|
Warning Review limit reached
Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe OSS readback retry flow now allows 60 attempts within a 600-second budget. Backoff sleeps are capped by the remaining budget, and the loop skips sleeping when the budget is exhausted. ChangesOSS readback retries
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust_hft/tools/collector/src/polymarket_upload.rs`:
- Around line 68-71: Update the readback retry flow around budget_remaining and
download_remote_artifacts_with to use one absolute deadline, recomputing the
remaining duration before each data, manifest, and success download and skipping
attempts with no time left. Treat elapsed time equal to
OSS_READBACK_MAX_WALL_CLOCK as expired before starting an attempt, while
preserving the existing sleep clamp. Add a deterministic counterexample test
covering the exact-deadline boundary and sequential-download budget enforcement.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: bba85ee7-520e-4158-904c-968f5feb9266
📒 Files selected for processing (1)
rust_hft/tools/collector/src/polymarket_upload.rs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d0e186a8f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
CodeRabbit round on #575: (1) per-file timeouts now consume the shared budget sequentially instead of each receiving the full remainder; (2) attempt count raised to 120 so the 600s deadline, not the count, bounds the loop; (3) the gate's real-market preflight budget is raised 300s -> 1200s so it cannot kill the wider uploader readback mid-flight. Refs #556
…577) * fix(deploy): widen gate-script OSS readback retry to 30x20s (~600s) Production evidence (2026-08-01T20:31:07+08, gate invocation 55808db5): a 117MiB multipart shadow object became HEAD-able exactly 168s after publication began — one second past the fifteenth 12s retry. The uploader-side budget (600s, #575) already covers this; the gate script's own readback did not. Refs #556 * fix(deploy): clamp the 20s readback backoff to the caller deadline CodeRabbit on #577: with 14-19s remaining the 13s guard permitted a 20s sleep past the deadline. Refs #556 --------- Co-authored-by: Sonic Shih <sonic.shih@mandonothing.com>
Change contract
Uploader readback-after-upload: up to 60 attempts bounded by a 600s wall-clock budget (budget-clamped sleeps, per-file timeout stays 60s), replacing the 12×5s loop.
Evidence
Fourth identical failure, 2026-08-01T18:23 CST (gate invocation 8d986d26): a 109MiB shadow upload (27-part multipart) took ~150s to become HEAD-able —
Last-Modifiedequals the final 404 second, exactly as in the three earlier cases. The 12×5s loop (~90s) ran out mid-publication. Every observed failure has been inside the uploader's own post-PUT verification (the.upload-staging/.oss-verifypath), never the gate script's readback — this PR targets the layer that actually fails.Out of scope
Dependency / merge order
None. Supersedes the retry constants from #565.
Focused validation
cargo test -p hft-collector --lib polymarket_upload --locked: 62 passed, 0 failed, ~1sRollout / rollback impact
Retry timing only; fail-closed semantics unchanged. Rollback = revert.
Issue relationship
Refs #556
Summary by CodeRabbit