fix(collector): widen uploader OSS readback retry window

[proerror77]

Change contract

Uploader readback-after-upload: 12 attempts × 5s backoff, per-file timeout capped at 60s, and a 300s wall-clock budget for the whole loop (replacing the 3×1s window). This is the actual root cause of the repeated shadow-gate failures on 2026-08-01: the gate-script retry patch (#556) could not help because the 404 happens inside the uploader binary's own post-PUT verification.

Production evidence

Three gate invocations (3736e02d, 842bab38, 6d1bfe7f) failed on ossutil cp 404 NoSuchKey of just-uploaded shadow objects; in each case the object became HEAD-able moments after the 3×1s window expired (verified via ossutil stat afterwards). With the wider window the readback succeeds instead of failing the upload (and thus the gate).

Second-round review fixes

• The first revision's 120s backoff cap was dead code (11×5s sleeps = 55s max); replaced with a wall-clock budget covering command time + sleeps.
• Downloads previously inherited the 300s upload timeout (36 × 300s worst case); per-file readback timeout now capped at 60s.
• Retry delay is 5ms under cfg(test): suite runs in 0.69s (was 55.7s).

Out of scope

• Gate-script readback retry (done in fix(deploy): retry shadow-gate OSS readback on transient publication-race 404 #556, complementary layer)
• Root cause on the OSS side (endpoint visibility lag); publish-order change (_SUCCESS/data-last) to be considered separately

Dependency / merge order

None. Release note: candidate cb55ac1d carries the old 3×1s window and cannot pass the readback in this environment — the next release built after this merge must be gated instead.

Focused validation

• cargo test -p hft-collector --lib polymarket_upload --locked: 62 passed, 0 failed, 0.69s

Rollout / rollback impact

Retry timing only; fail-closed semantics unchanged (upload still fails if the object never appears). Rollback = revert.

Issue relationship

Refs #556

Summary by CodeRabbit

• Bug Fixes
  • Improved reliability when reading back uploaded files from storage.
  • Added automatic retries and time limits to prevent stalled downloads.
  • Ensured each file download respects the configured timeout limits.

[coderabbitai]

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 17 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1cc5283d-5c9c-4ffb-a5bd-68c0e9e52891

📥 Commits

Reviewing files that changed from the base of the PR and between 339b76f and 74eb889.

📒 Files selected for processing (1)

• rust_hft/tools/collector/src/polymarket_upload.rs

📝 Walkthrough

Walkthrough

OSS readback now retries with bounded attempts and delays. Each artifact download uses a capped timeout. The retry loop also enforces a total wall-clock limit.

Changes

OSS readback control

Layer / File(s)	Summary
Bound OSS readback retries rust_hft/tools/collector/src/polymarket_upload.rs	The collector increases retry attempts, applies production and test delays, caps each download at 60 seconds, and stops after 300 seconds or the attempt limit.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• proerror77/monday#34: Introduced related OSS readback logic in the same collector file.
• proerror77/monday#376: Modified the same OSS readback verification retry logic.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the widened uploader OSS readback retry window.
Description check	✅ Passed	The description covers the change, issue relationship, scope, dependencies, validation, rollout, rollback, and fail-closed behavior.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/upload-readback-window

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c7c64f011c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust_hft/tools/collector/src/polymarket_upload.rs`:
- Line 1886: Update download_remote_artifacts_with and its retry loop so every
runner invocation and retry delay is capped by the remaining
OSS_READBACK_MAX_WALL_CLOCK deadline; pass the deadline into the function, stop
immediately when no duration remains, and preserve existing retry behavior
otherwise. Add a deterministic regression test covering exhaustion during
sequential downloads or retry delay.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 19d767ac-25fa-4a86-b0d8-a71f0c49df43

📥 Commits

Reviewing files that changed from the base of the PR and between 3ed335b and 339b76f.

📒 Files selected for processing (1)

• rust_hft/tools/collector/src/polymarket_upload.rs