diff --git a/deployment/aliyun/README.md b/deployment/aliyun/README.md index 7cdc00d36..f7b73477f 100644 --- a/deployment/aliyun/README.md +++ b/deployment/aliyun/README.md @@ -280,6 +280,27 @@ DingTalk webhook robot, and recovery notification must be enabled so a resolved incident is visible. Cloud Monitor is the PRIMARY channel because it fires even when every GitHub and repository path is down. +Live alarm identity (created 2026-08-08 via `PutResourceMetricRule`, +readback-verified with `DescribeMetricRuleList`; both rules enabled, +`AlertState: OK`, recovery notification on): + +- `monday-collector-disk-warn` — `acs_ecs_dashboard/diskusage_utilization`, + instance `i-6we6afeqsvv8uo1ixmyo`, Average > 75% for 3 consecutive 60s + periods, Warn level, contact group `云账号报警联系人`. +- `monday-collector-disk-critical` — same metric and scope, Average > 90% for + 3 consecutive 60s periods, Critical level, same contact group. + +Deviation from the representative JSON above: period 300 is not a supported +alarm period for `diskusage_utilization` (the metric supports 15/60/900), so +the live rules evaluate 60s × 3 consecutive periods. + +Open gaps: (1) the CloudMonitor guest agent does not report +`diskusage_utilization` for this instance (zero datapoints in the 7 days before +2026-08-08), so the alarms cannot fire until the agent is installed on the +host; (2) the `monday-oncall` contact group with email + DingTalk webhook does +not exist yet — the rules currently notify only `云账号报警联系人` and have no +webhook. + ### GitHub Actions workflow (fallback) `.github/workflows/monitor-collector-host.yml` runs every 15 minutes. It diff --git a/docs/reports/2026-08-08-polymarket-parity-high-rate-tapes-known-issue.md b/docs/reports/2026-08-08-polymarket-parity-high-rate-tapes-known-issue.md new file mode 100644 index 000000000..17d235d4c --- /dev/null +++ b/docs/reports/2026-08-08-polymarket-parity-high-rate-tapes-known-issue.md @@ -0,0 +1,211 @@ +--- +name: polymarket-parity-high-rate-tapes-known-issue +description: Known issue — polymarket shadow parity validator cannot validate high-rate tapes (stable-read race on the live baseline spool), with follow-up fix proposal +created: 2026-08-08T01:15:00Z +updated: 2026-08-08T01:15:00Z +status: open +--- + +# Known Issue: Polymarket Parity Validator Fails on High-Rate Tapes + +**Follow-up issue:** https://github.com/proerror77/monday/issues/747 +(`bug`, `needs-triage`) + +## Summary + +The Polymarket shadow parity validator +(`rust_hft/tools/collector/src/polymarket_parity.rs`, invoked as +`polymarket-raw-ops verify-shadow-parity` from +`rust_hft/tools/collector/src/bin/polymarket-raw-ops.rs:319-339`) validates the +Rust shadow collector against the live baseline lane by requiring **every tape +in both spools to remain completely unchanged while it is read twice**. The +shadow gate stops and finalizes only the shadow lane before verification; the +baseline lane keeps collecting into its spool throughout. On high-rate +(high-frequency) market tapes the baseline appends or rotates during a read +pass, the stability check aborts the pass, the fixed retry budget exhausts, and +the verifier bails with `spool changed while reading parity window`. The gate +then fails closed with **no parity evidence at all**: byte, field, dedupe, +settlement, and rotation parity are unvalidated for that window, and candidate +promotion is blocked for operational (not correctness) reasons. + +This is a structural defect, not a tuning problem; see +[Why this is structural](#why-this-is-structural). + +## Evidence + +### Primary defect: all-or-nothing stable-read of a live spool + +- `FileFingerprint` (`polymarket_parity.rs:79-98`) captures device, inode, + byte size, and mtime to nanosecond precision. Any append to a tape — even + rows far outside the comparison window — changes the fingerprint. +- `stream_stable_rows` (`polymarket_parity.rs:236-289`) compares the + fingerprint before opening (`:246-247`), after opening (`:250-251`), at end + of file (`:285-286`), and around every row error (`:272-274`, `:279-281`); + a missing trailing newline also returns "changed" (`:265-266`). Any change + yields `Ok(None)`, i.e. "tape moved, start over". +- `load_rows` (`polymarket_parity.rs:345-419`) reads **all** tapes in the + spool in a first pass (`:358`), re-reads all of them in a second metadata + pass (`:385`), and re-enumerates the directory between passes (`:371`, + `:406`). The whole two-pass cycle is retried at most 5 times with 20 ms + sleeps (`:352`, `:373`, `:407`) and then fails closed + (`:418`: `bail!("{}: {last_reason}", ...)` with + `last_reason = "spool changed while reading parity window"`). +- The gate stops, freezes, and finalizes only the **shadow** collector + (`deployment/aliyun/polymarket-raw-ops-shadow-gate.sh:1887-1916`), then runs + the verifier against `--legacy-spool "$LEGACY_SPOOL"` (`:1922-1926`) while + `LEGACY_RUNTIME_STABILITY_REQUIRED=true` (`:25`) requires the baseline unit + to be active and healthy for the entire gate + (`verify_runtime_identity`, `:399-417`, checks `systemctl is-active`). The + baseline spool is therefore a live, continuously appended directory at + verification time — by design of the gate, not by accident. +- The gate policy requires all parity checks true and non-empty legacy metrics + in `legacy_overlap` mode + (`deployment/aliyun/polymarket-shadow-gate-policy.jq:329-337`, `:370-373`), + so a verifier bail is never masked: it always fails the gate. + +Per-attempt failure probability is the probability that at least one append or +rotation lands anywhere in a two-pass read of the whole spool. As tape rate +rises, the mean inter-append interval falls below the pass duration and that +probability approaches 1; 5 retries do not change the asymptote. + +### Amplifier A: read cost scales with total spool bytes, not the window + +`tape_paths` (`polymarket_parity.rs:162-189`) enumerates every closed segment +plus the active tape, and both passes parse every row of every tape as JSON; +`retain_primary_row` (`:291-343`) discards out-of-window rows only **after** +the full parse. Rows appended after the comparison cutoff are likewise parsed +and held to full schema validity (kind allowlist `:226-229`, gapless sequence +`:205-218`), so a post-cutoff row the running baseline has written but the +verifier's schema predates is a hard error, not a retry. On a host with upload +backlog — exactly the condition of the 2026-08-05/06 disk-full incident +(`docs/reviews/2026-08-07-incident-remediation-diff-review.md`) — spool +retention is large, each pass is slow, and the race window per attempt widens +further. + +### Amplifier B: memory scales with tape rate + +Every in-window row is retained in a `Vec` (`:359-361`, extended at +`:410`), and `trade_map` clones every trade's full `serde_json::Value` into a +`BTreeMap` (`:668`); metadata and settlement maps behave similarly +(`:569-596`, `:704-738`). Peak verifier memory is O(in-window rows × row size) +on the 7.75 GiB collector host +(`docs/reports/polymarket-shadow-memory-calibration-2026-07-16.md`), and the +verifier runs inline in the gate shell (`polymarket-raw-ops-shadow-gate.sh:1932`) +rather than under the shadow unit's memory envelope. On sufficiently high-rate +windows the verifier can be OOM-killed before producing evidence even when the +read race is won. + +### Ranked candidates considered + +1. **Stable-read race on the live baseline spool (structural, primary).** + Breaks validation outright on high-rate tapes. Detailed above. +2. **O(spool) read cost and post-cutoff full-schema parsing (structural, + amplifier).** Same root design — the read scope is "the whole spool, both + passes" instead of "the comparison window" — so it is fixed by the same + redesign, not separately. +3. **O(window) in-memory retention with full value clones (structural, + secondary).** Degrades rather than breaks: OOM-kill on very high-rate + windows. Fixed by streaming/digest comparison; independently shippable. +4. **Fixed 600 s trade maturity lag / 601 s gate tail (incidental).** + `TRADE_MATURITY_LAG_SECONDS` (`polymarket_parity.rs:37`) and + `PARITY_TAIL_SECONDS` (`polymarket-raw-ops-shadow-gate.sh:12`) are policy + constants about provider eventual consistency, not tape rate. A tuning + knob, not the defect. +5. **Within-lane duplicate `record_id` fails `dedupe_parity` + (`polymarket_parity.rs:667-673`, `:1042-1048`).** Reviewed and judged + fail-closed by design: collectors keep `trade_seen` state and must dedupe; + a duplicate is genuine collector evidence, not a validator artifact. + +## Impact + +On high-rate tapes the shadow gate loses **all** parity guarantees for the +affected window — byte identity, field presence, dedupe, trade coverage and +contract, settlement, and rotation — because the verifier cannot complete a +stable read of the live baseline spool and bails before writing evidence. The +failure is fail-closed, so there is no false promotion risk; the cost is that +candidate releases cannot be promoted while baseline tape rates are high (or +while spool retention is large after an upload backlog), and gate failures +present as operational noise (`spool changed while reading parity window`) +rather than as a validator limitation. The break threshold is reached when the +baseline lane's append/rotation interval approaches the two-pass full-spool +read duration; at current reference-lane rates (7 symbols, 5/15-minute +markets) the race is rare, which is why it has not been observed in +production. This is a code-level structural analysis with a deterministic +synthetic reproducer (appending writer fixture); it has **not** been observed +against a live high-rate tape, and no production gate failure is attributed to +it here. + +## Why this is structural + +The validator's core protocol is "read the entire spool twice and require the +world not to move". No constant tuning removes the race: + +- More retries or longer sleeps only multiply an attempt whose failure + probability already tends to 1 as tape rate grows. +- The gate contract itself keeps the baseline running + (`LEGACY_RUNTIME_STABILITY_REQUIRED=true`), so the writer cannot be + quiesced without weakening an independent gate guarantee. +- Pass duration grows with spool size (Amplifier A), so the race gets worse + exactly when the system is under stress (upload backlog, high-rate tapes). + +The defect is in the read model — global quiescence over unbounded scope — +not in any threshold. The fix must change what is read and how stability is +judged, which is a redesign of `load_rows`/`stream_stable_rows`, not a +parameter change. + +## Proposed follow-up fix direction + +One PR, one behavior, scoped to the validator: + +1. **Append-tolerant stability.** Tapes are append-only; treat growth as safe. + Snapshot `(device, inode, size)` at pass start, read only bytes below the + snapshot size, and afterwards verify identity is unchanged + (same device/inode, size ≥ snapshot, no rename) instead of requiring equal + size and mtime. Truncation, replacement, and indirection keep failing + closed exactly as today. +2. **Window-scoped reads.** Select candidate tapes by rotation timestamp/name + and mtime overlap with the comparison window before parsing, and stop + parsing a tape once `recorded_at` exceeds the cutoff (rows are + time-ordered per tape). Read cost becomes O(window), decoupling gate + latency from spool retention. +3. **Do not impose full schema validity on rows outside the comparison + window.** Parse enough to sequence-check and skip them; only in-window + rows get contract validation. This removes the post-cutoff hard-error + exposure described in Amplifier A. +4. **Bounded memory (secondary, may be a stacked PR).** Replace retained + `Vec` + cloned `BTreeMap`s with a streaming per-lane digest/set + comparison (sorted `record_id` merge or per-identity hash accumulation), + so peak memory no longer scales with in-window trade count. + +Fail-closed semantics are the contract and must not move: sequence gaps, +truncation, symlink/indirection, schema violations inside the window, and +duplicate identities must still fail; the `monday.polymarket_shadow_parity.v1` +evidence metrics consumed by `polymarket-shadow-gate-policy.jq` stay +compatible, or schema and policy are versioned together in the same change. + +## Acceptance criteria for the follow-up PR + +- New focused test: a writer appends valid rows to the legacy spool's active + tape at ≥ 100 rows/second while `compare()` runs; verification succeeds + within the existing attempt budget and produces passing evidence. +- New focused test: a spool containing large out-of-window segments plus a + small in-window set completes with rows-parsed / wall-time bounded by the + window, not the spool size (assertion on parsed-row counts or a deterministic + fixture bound). +- Regression tests proving fail-closed behavior is unchanged: mid-read + truncation, rename/replacement, symlink swap, sequence gap, in-window schema + violation, and duplicate `record_id` still error or fail parity. +- All existing tests in `rust_hft/tools/collector/src/polymarket_parity.rs` + pass unchanged (`cargo test -p hft-collector --locked`, scoped to the + collector crate per `AGENTS.md`), plus scoped Clippy clean. +- Evidence schema: `monday.polymarket_shadow_parity.v1` metrics consumed by + the gate policy are unchanged, or any change ships with the matching + `polymarket-shadow-gate-policy.jq` update in the same PR. +- No changes to gate recover logic (owned by a separate follow-up) and no + changes to collector write paths. + +## Out of scope + +Gate recover logic and its own write-up are owned by another follow-up item +and are intentionally untouched here. Production runtime mutation is not part +of this document; no cloud or host state was changed while preparing it. diff --git a/docs/reports/2026-08-08-polymarket-raw-ops-gate-recover-known-issue.md b/docs/reports/2026-08-08-polymarket-raw-ops-gate-recover-known-issue.md new file mode 100644 index 000000000..57f9ec324 --- /dev/null +++ b/docs/reports/2026-08-08-polymarket-raw-ops-gate-recover-known-issue.md @@ -0,0 +1,259 @@ +--- +name: polymarket-raw-ops-gate-recover-known-issue +description: Known issue — polymarket raw-ops gate recover admission conflates systemd bookkeeping state with containment, refusing failed-state units with no governed remediation, with follow-up fix proposal +created: 2026-08-08T01:18:36Z +updated: 2026-08-08T01:18:36Z +status: open +--- + +# Known Issue: Polymarket Raw-Ops Gate `recover` Refuses Failed-State Contained Units + +**Follow-up issue:** https://github.com/proerror77/monday/issues/748 +(`bug`, `needs-triage`) + +## Summary + +The `recover` action of the Polymarket raw-ops gate control plane +(`deployment/aliyun/polymarket-raw-ops-gate-control.sh:727-738`, added by #637 / +PR #644, retargeted to the healthy Gamma closed-200 probe by PR #717) admits a +recovery gate only when the contained baseline collector and all four +uploader units/timers report systemd `ActiveState` exactly `inactive` +(`polymarket-raw-ops-gate-control.sh:676-679` and `:717-725`). A unit in +`failed` state — no managed process, restart budget exhausted or `Restart=no`, +i.e. fully contained — is refused with `recovery requires the direct bootstrap +baseline to be stopped` or `recovery requires inactive uploader/timer`. + +`failed` is the state the production units are most likely to occupy in +exactly the conditions `recover` was built for: the action's own contract +(#637) is "when, and only when, the direct Rust bootstrap reference collector +has been contained after the known closed-lane Gamma tagged-500 failure", and +the 2026-08-05/06 disk-full incident +(`docs/reviews/2026-08-07-incident-remediation-diff-review.md`) left ample +opportunity for crash-loop exhaustion and failed oneshot uploaders (the +pre-#5 `NoSuchKey` upload failures). The control plane offers no governed way +forward from that refusal: the only remediation is an unrecorded manual +`systemctl reset-failed`/`systemctl stop` on production units outside the +control lock — or, worse, manual stop/start experimentation on the production +collector to reach `inactive`, an ungoverned runtime transition with no health +verification during an incident. + +The defect is in the admission predicate, not in the safety posture: the gate +remains fail-closed throughout, so there is no false-promotion risk. See +[What is sound](#what-is-sound-verified) and +[Why this is a defect](#why-this-is-a-defect-not-a-missing-enhancement). + +## Evidence + +### Primary defect: `ActiveState == "inactive"` is used as the containment test + +- Baseline admission (`polymarket-raw-ops-gate-control.sh:676-682`): + `recovery_baseline` dies unless `ActiveState` is exactly `inactive`, then + separately requires `MainPID == 0`. The containment fact is already + established by `MainPID == 0` plus the exact identity binding that follows + (fragment `:683-686`, drop-ins `:687-689`, effective ExecStart `:690-693`, + restart counter and invocation ID `:694-700`, direct secure binary and + digest `:701-707`). The `inactive` string adds no safety — it selects a + systemd bookkeeping state, not a runtime property. A `failed` unit has no + managed process and systemd will not start it on its own (the restart + budget is exhausted or `Restart=no`), which is why `systemctl is-active + --quiet` exits non-zero for it. +- Uploader admission (`polymarket-raw-ops-gate-control.sh:717-725`): + `verify_recovery_uploaders_stopped` applies the same `inactive`-only test to + `polymarket-reference-upload.service`/`.timer` and + `polymarket-market-tape-upload.service`/`.timer`. A failed oneshot uploader + — the common post-incident residue — is contained but refused. +- The downstream machinery binds the *recorded* snapshot, not the `inactive` + constant: `verify_contained_recovery_baseline` in the gate + (`deployment/aliyun/polymarket-raw-ops-shadow-gate.sh:524-551`) compares the + live `ActiveState` to the snapshot value (`:529-531`), and the cutover's + `verify_contained_bootstrap_recovery` + (`deployment/aliyun/polymarket-raw-ops-cutover.sh:955-997`, live comparison + at `:976-979`, invoked at `:1689` and again at `:1849`) does the same. The + literal `"inactive"` appears only in binding predicates — gate + `verify_recovery_binding` (`polymarket-raw-ops-shadow-gate.sh:501`), gate + policy `contained_bootstrap_recovery` + (`deployment/aliyun/polymarket-shadow-gate-policy.jq:80`), and cutover + (`polymarket-raw-ops-cutover.sh:964`) — so the recovery chain is + mechanically compatible with any recorded quiescent state; the refusal is an + admission-time artifact. +- The harness pins the intended semantics + (`deployment/aliyun/test-polymarket-raw-ops-control-plane.sh:434-488`): an + active baseline, active uploader, wrong ExecStart, stale/missing/wrong + candidate probe are all rejected, and recovery must never + start/stop/restart/enable/disable the contained baseline (`:481-488`). It + never exercises a `failed`-state baseline or uploader, so the refusal of the + primary post-failure state is currently untested in either direction. + +### Secondary gap A: admission preconditions run outside the control lock and leave no failure evidence + +`recover_gate` (`polymarket-raw-ops-gate-control.sh:727-738`) evaluates the +probe, baseline, and uploader preconditions *before* `start_gate` acquires +`CONTROL_LOCK` (`:764-765`). The window is closed downstream — the gate +re-verifies admission freshness and baseline identity at start +(`polymarket-raw-ops-shadow-gate.sh:1433-1438`) and repeatedly through the +gate (`:1663`, `:1693`, `:1871`, `:1956`, `:2111`), and the cutover re-verifies +twice — so no unsafe promotion can result. But a refused or interleaved +admission produces nothing durable: `die` writes only to stderr. During +incident response — the only time `recover` runs — there is no immutable +record of admission attempts, refusals, or their reasons. The governed-restore +precedent takes the host-wide locks *first* +(`deployment/aliyun/host-rust-lob-restore.sh:517-526`) and writes +`recovery.json` with `result=failed`, the failing step, and the reason on +every failure path (`host-rust-lob-restore.sh:202-241`, `:306-322`). + +### Secondary gap B: baseline identity is bound from systemctl-loaded values only + +`recovery_baseline` reads `FragmentPath`/`ExecStart`/counters via +`systemctl show`, i.e. the unit as loaded at the last `daemon-reload`; the +on-disk fragment bytes are never compared (contrast the restore bar, which +`cmp`s every installed unit/env asset against the gated bundle, +`host-rust-lob-restore.sh:444-456`). A unit file edited without a +`daemon-reload` would be bound into the recovery evidence in its stale loaded +form. This is a residual risk, not a live hole: promotion is neutralized +because the cutover installs its own unit assets atomically rather than +inheriting the stale file, and any `daemon-reload` before gate start surfaces +the drift at the in-gate identity check. + +## What is sound (verified) + +The recover path's fail-closed core is intact; this document does not allege a +safety defect: + +- Probe handling: canonicalization and containment under the exact + per-candidate evidence root (`polymarket-raw-ops-gate-control.sh:647-650`), + schema-exact binding to candidate/source with the bounded Gamma closed-200 + contract (`:652-662`), and a 900-second freshness budget enforced both at + admission (`:663-668`) and again at gate start + (`polymarket-raw-ops-shadow-gate.sh:514-522`, called at `:1433-1435`). +- Baseline binding: direct, root-owned, non-symlink, executable binary with + digest distinct from the candidate + (`polymarket-raw-ops-gate-control.sh:701-707`); candidate binary digest + re-verified in `start_gate` (`:747-750`). +- In-gate containment: uploader inactivity re-checked at every baseline + identity verification (`polymarket-raw-ops-shadow-gate.sh:547-550`), which + runs at gate start and repeatedly through the gate (`:1437`, `:1663`, + `:1693`, `:1871`, `:1956`, `:2111`). +- Evidence handling: per-invocation immutable receipts and pass markers + serialized through `commit.lock` with staged/committed states + (`polymarket-raw-ops-gate-control.sh:405-485`); a passed recovery gate + embeds the recovery evidence in `gate.json`, which the policy pins via + `recovery_matches_gate` (`polymarket-shadow-gate-policy.jq:92-95`, `:294`). +- Cutover: contained-recovery promotion re-verifies the baseline twice and + rolls back transactionally to the recorded stopped state + (`polymarket-raw-ops-cutover.sh:1128-1136`, `:1279-1299`). + +## Impact + +- **Operational:** after a crash-loop or failed-oneshot containment — the + modal post-incident state — `recover` cannot admit a gate. The only paths + forward are ungoverned: manual `systemctl reset-failed`/`stop` on production + units outside `CONTROL_LOCK` with no evidence artifact, or manual stop/start + of the production collector to force `inactive`, an unrecorded production + runtime transition with no health verification, taken under incident + pressure. These are exactly the untracked host mutations the control plane + exists to eliminate and that the 2026-08-07 monitoring remediation (#7, PR + #735) now watches for. +- **Safety:** none. Every refusal is fail-closed; the defect blocks admission, + it cannot weaken a gate, skip a probe check, or promote a candidate. +- **Evidence:** refused admissions are invisible after the fact (stderr only), + so incident reviews cannot reconstruct recovery-gate admission history. + +## Relation to the governed-restore precedent + +`deployment/aliyun/host-rust-lob-restore.sh` (merged 2026-08-07, PR #734) is +the bar for this class of recovery action, and the recover path currently +falls short of it in three specific ways: + +1. **Quiescence test.** Restore treats any not-active unit as quiescent via + `systemctl is-active --quiet` (`host-rust-lob-restore.sh:425-428`), which + accepts `failed`; recover pins the literal string `inactive`. +2. **Governed remediation.** Restore performs `systemctl reset-failed` as a + named STEP inside the governed flow (`host-rust-lob-restore.sh:472`) before + starting; recover has no remediation step at all. +3. **Serialization and evidence.** Restore holds the host release lock and the + shadow-gate lock before any preflight (`host-rust-lob-restore.sh:517-526`) + and writes immutable evidence on success *and* failure + (`host-rust-lob-restore.sh:202-241`, `:306-322`); recover checks + preconditions before its lock and records nothing on refusal. + +## Why this is a defect, not a missing enhancement + +`recover` already has one job at admission: prove the baseline is contained +and exactly identified. The containment fact it needs is "no managed process +plus exact identity", and it already verifies that (`MainPID == 0`, fragment, +drop-ins, ExecStart, binary digest). The `ActiveState == "inactive"` test is +therefore not a missing layer of rigor — it is the wrong predicate, selecting +a bookkeeping state that the modal post-failure scenario does not produce, +while every downstream consumer merely requires that the recorded state match +the live state. An action that cannot be lawfully invoked in the conditions +its own contract (#637) scopes it to, and whose refusal pushes operators +toward ungoverned production mutations, is broken behavior with a focused +reproduction, not a feature request. + +## Proposed follow-up fix direction + +One PR, one behavior, scoped to the recover admission path: + +1. **Admit containment, not bookkeeping state.** Accept `inactive` or `failed` + for the baseline collector and the four uploader units/timers, keeping + `MainPID == 0` and every exact-identity check unchanged. +2. **Governed reset, restore-style.** For units observed `failed`, run + `systemctl reset-failed` as a governed step, then re-read every snapshot + field (state, MainPID, fragment, drop-ins, ExecStart, restarts, invocation, + binary digest) so the recorded baseline snapshot reflects the post-reset + state (`inactive`). The downstream binding predicates + (`polymarket-raw-ops-shadow-gate.sh:501`, + `polymarket-shadow-gate-policy.jq:80`, + `polymarket-raw-ops-cutover.sh:964`) then keep their exact `"inactive"` + contract unchanged — preferred over widening four binding sites to accept + `"failed"`, which would also stay racy against any manual reset between + admission and gate start. +3. **Serialize admission.** Acquire `CONTROL_LOCK` at the top of + `recover_gate` so precondition reads, the governed reset, and the gate + start are one critical section (mirroring + `host-rust-lob-restore.sh:517-526`). +4. **Durable admission evidence.** Write an immutable admission record + (accepted/refused, exact candidate/baseline/probe identities, refusal + reason) under the gate evidence root on every `recover` invocation, + mirroring `recovery.json` (`host-rust-lob-restore.sh:202-241`). +5. **Optional, stacked:** compare the on-disk baseline unit fragment bytes + against the expected installed asset (or `daemon-reload` and re-read) + before binding, closing secondary gap B. + +Fail-closed semantics must not move: active/activating/deactivating units, +nonzero MainPID, any identity drift, active uploader/timer, and stale, +missing, or mis-bound probes must keep refusing. + +## Acceptance criteria for the follow-up PR + +- New harness cases in + `deployment/aliyun/test-polymarket-raw-ops-control-plane.sh`: a `failed` + baseline and each `failed` uploader unit are admitted after a governed + `reset-failed`; the recorded baseline snapshot reads `inactive` with + `MainPID == 0`; the gate binding, gate policy, and cutover recovery binding + pass unchanged end-to-end on the existing recovery evidence path. +- Regression: `active`, `activating`, and `deactivating` baselines, active + uploaders/timers, and every existing refusal case (probe stale/missing/wrong + candidate, ExecStart/fragment/drop-in/binary drift, active gate for the same + candidate) still refuse; the existing assertion that recovery never + start/stop/restart/enable/disables the contained baseline + (`test-polymarket-raw-ops-control-plane.sh:481-488`) is preserved and + extended to prove `reset-failed` happens only inside the control lock. +- Admission evidence: both an admitted and a refused `recover` invocation + write an immutable record with exact identities and reason, asserted in the + harness. +- `deployment/aliyun/test-polymarket-raw-ops-control-plane.sh` passes in full; + `shellcheck` and `bash -n` clean on touched scripts; `git diff --check` + clean. +- Boundaries: no changes to the parity validator (owned by issue #747 and its + own write-up); no weakening of probe freshness, identity binding, or + `MainPID == 0` containment; no cloud or host mutation — runtime application + of any fix remains a separately authorized cutover. + +## Out of scope + +The parity validator defect and its write-up are owned by a separate +follow-up (issue #747, +`docs/reports/2026-08-08-polymarket-parity-high-rate-tapes-known-issue.md`). +Production runtime mutation is not part of this document; no cloud or host +state was changed while preparing it.