feat(deploy): add Polymarket tape upload watchdog timer - #657
Conversation
Add polymarket-market-tape-upload-watchdog.{sh,service,timer}: a
two-minute oneshot watchdog that self-heals both Polymarket upload
lanes after the market upload timer sat silently inactive for 19
hours and the reference upload timer was later found dead as well.
Per run it logs one journal INFO line (tag polymarket-upload-watchdog)
with each lane's pending rotated-tape count, oldest rotated-tape age,
and /data free GB; starts an inactive upload timer with a WARNING
carrying the previous state; and starts an inactive upload service
with --no-block when a rotated tape is older than 90 minutes. It only
ever starts units and never modifies tape files. The unit runs as root
for systemctl start but keeps the remaining hardening of the existing
upload units. README documents installation alongside the upload units.
|
Warning Review limit reached
Next review available in: 13 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e559042aa3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
All three P1 threads addressed in
Stub suite: 7/7 scenarios pass; |
Change contract
On 2026-08-03 the market-tape upload timer was found silently stopped for 19 h (backlog 145 GB, disk one hour from full), and the reference-upload timer was found dead the same way. This adds a small root oneshot + 2-minute timer (
polymarket-market-tape-upload-watchdog.{sh,service,timer}) that self-heals both lanes: an inactivepolymarket-market-tape-upload.timerorpolymarket-reference-upload.timeris restarted with a WARNING log; an inactive/failed service with rotated tapes older than 90 min gets a--no-blockstart; every run logs one INFO line with per-lane pending count, oldest tape age, and/datafree GB. It never stops, disables, or deletes anything.Issue relationship
Refs #655
Out of scope
monday-trade-data-26(runtime action, needs the named-controller procedure; README documents the install snippet).Dependencies and merge order
None
Focused validation
sh -n/bash -nOK;shellcheck -s shexit 0.systemctl/logger/dfon PATH, real script undersh): all 5 two-lane scenarios pass, including counterexamples — a dead timer infailedstate is healed, a fresh tape (<90 min) triggers no remediation, and the activemarket-updates.ndjsonis never counted or touched.deployment/aliyun/test-polymarket-raw-ops-control-plane.sh: passed (run with GNU coreutils shimmed on macOS; README assertions re-verified directly after the reference-lane extension).systemd-analyze verifynot available on macOS — will be run on the host before install (noted in PR).Rollout and rollback
No production impact until installed per the README snippet; rollback is
systemctl disable --now polymarket-market-tape-upload-watchdog.timerand removing the three files.Scope exception
None