fix(monitor): hard-gate collector health on the upload delivery contract

.github/workflows/monitor-collector-host.yml

@@ -193,6 +193,7 @@ jobs:
           set -euo pipefail
           breaches=$(jq -r '.breaches[]' /tmp/collector-health.json)
           checked_at=$(jq -r '.checked_at' /tmp/collector-health.json)
+          warnings=$(jq -r '(.warnings // [])[]' /tmp/collector-health.json)
           {
             printf 'Host: monday-trade-data-26 (Aliyun Tokyo ap-northeast-1, instance %s)\n' "${{ env.INSTANCE_ID }}"
             printf 'Checked at: %s\n' "$checked_at"
@@ -201,6 +202,10 @@ jobs:
             printf 'Primary alert channel is the Cloud Monitor disk alarm; this issue is the repository fallback and needs human triage.\n\n'
             printf 'Breaches:\n'
             printf '%s\n' "$breaches" | sed 's/^/- /'
+            if [ -n "$warnings" ]; then
+              printf '\nWarnings (not blocking):\n'
+              printf '%s\n' "$warnings" | sed 's/^/- /'
+            fi
           } > /tmp/collector-health-issue.md
 
           existing=$(gh issue list --state open --search 'in:title "Durable monitor breach"' \

agent-worktree.yml

@@ -1,10 +1,13 @@
 owner: codex
-contract: actionlint-macos-15-intel
-worktree: /Users/proerror/Documents/monday/.worktrees/codex/actionlint-macos-15-intel
-branch: codex/actionlint-macos-15-intel
-base_sha: 161b761dff56acb758dd4a1787fd0b3b8130196a
+contract: collector-health-hard-gates
+worktree: /Users/proerror/Documents/monday/.worktrees/codex/collector-health-hard-gates
+branch: codex/collector-health-hard-gates
+base_sha: be9e746a1ddf7e6f26264c4811c4a3511c3caa87
 allowed_files:
-  - .github/workflows/ploy-ci.yml
   - agent-worktree.yml
-dependency: none
+  - deployment/aliyun/monday-collector-health.sh
+  - deployment/aliyun/test-monday-collector-health.sh
+  - .github/workflows/monitor-collector-host.yml
+  - deployment/aliyun/README.md
+dependency: upload-status last_success_at for the fee uploader lands in a parallel PR; this contract treats a missing last_success_at as a breach, so it must merge after or together with that PR
 no_external_runtime_mutation: true

deployment/aliyun/README.md

@@ -162,19 +162,34 @@ tape files or `upload-status.json`. It emits one JSON snapshot (or a human
 to journald tag `monday-collector-health`. Run with `--json` for machine output
 and `--dry-run` to avoid reading or writing the persistent delta state.
 
-| Check | Breach condition |
+The monitor has exactly four hard gates. Each is a breach: it fails closed into
+the `monitor-collector-host` workflow issue and blocks `ok:true`.
+
+| Hard gate | Breach condition |
+| --- | --- |
+| 1. Status file | `upload-status.json` missing, a symlink, or unparseable on the mandated lanes (`binance-lob` spot/usdm, `binance-fee`) |
+| 2. Upload freshness | `last_success_at` missing/unparseable, or older than the lane bound (LOB 7200s, fee 600s, usdm-reference 1200s, polymarket 7200s, bybit 5400s; each just above the lane's upload cadence — the polymarket lanes rotate tapes hourly, so the 5-minute upload timer is not a heartbeat) |
+| 3. Pending backlog | pending count over the lane limit, or oldest pending artifact older than the lane age bound, using each collector's own pending definition (LOB `*.manifest.json`, fee/usdm-reference `lake/raw/**/batch=*`, polymarket rotated `market-updates.*.ndjson` tapes, bybit marked `.ndjson` without `.uploaded.json`) |
+| 4. Upload failures | `last_error_at`/`last_error` present, or a `failure_count` increase since the previous poll (prior counts live under `/var/lib/monday-collector-health`) |
+
+Two legacy guards stay breaches: `polymarket-raw-ops-gate@.service` must remain
+disabled/masked/not-found (`static` proves an uncleaned host installation), and
+state-persistence failures (gate 4 delta detection depends on that state).
+
+Every other check is a warning — reported in the JSON `warnings` array and as
+`warning:` lines, never blocking `ok:true`:
+
+| Warning | Condition |
 | --- | --- |
 | `/data` disk | free < 25% (warn) or < 10% (critical) via `df -Pk /data` |
-| Governed services | `binance-lob-archiver-production@spot/usdm` and `binance-usdm-reference-collector` active AND enabled AND `Result==success`, plus a restart-rate delta > 1 since the last poll |
-| Upload lanes | `polymarket-market-tape-upload.timer` and `polymarket-reference-upload.timer` active AND enabled; their oneshot services' last `Result==success` |
-| Watchdog | `polymarket-market-tape-upload-watchdog.timer` active AND enabled; the watchdog service's last `Result==success` |
-| Incident fill source | `bybit-options-archiver.service` and `polymarket-raw-ops-gate@.service` must stay disabled/masked/not-found |
+| Governed services | `binance-lob-archiver-production@spot/usdm`, `binance-usdm-reference-collector`, `bybit-options-archiver` active AND enabled AND `Result==success`, plus a restart-rate delta > 1 since the last poll |
+| Upload lane units | upload/watchdog/fee timers active AND enabled; their oneshot services' last `Result==success` |
 | `health.json` | missing/unparseable, wall-clock age of `updated_at_ns` > 300s, or `sequence_gaps` > 0 (spot + usdm spools) |
-| `upload-status.json` | `last_error_at`/`last_error` present, or a `failure_count` delta since the previous poll (prior counts live under `/var/lib/monday-collector-health`) |
 | Delay-gate trips | > 0 journald `source-to-receive delay exceeds the governed limit` lines per Binance unit in the last 15 minutes |
+| Fee snapshot failures | > 0 `Failed with result` journald lines per fee snapshot unit in the last 10 minutes |
 | `/data` mount | `mountpoint -q /data` fails (the monitor must DETECT a missing mount, not gate on it) |
 
-The persistent-service check deliberately does not breach on `NRestarts > 0`:
+The persistent-service check deliberately does not warn on `NRestarts > 0`:
 both Binance archivers restart every six hours by design
 (`RuntimeMaxSec=21600`). Crash loops are detected through `Result != success`
 or an `NRestarts` delta greater than one between consecutive five-minute polls.