Problem
Production usdm wedged silently for ~36h (2026-08-12 13:04Z → 08-14 00:41Z) in a recovery crash loop: a 4.7GB .part segment could not be recovered by recover_parts — it fs::reads the entire file into RAM (4.4G RSS peak, lob_archiver.rs:820), runs two full-file validation passes, then zstd with a fixed 300s timeout (lob_archiver.rs:687) that cannot finish on a pressured host → main() exits 1 → Restart=always → identical failure forever. One oversized poisoned part permanently blocked the whole lane and later blocked the V2 cutover drain.
Evidence: investigation on host i-6we6afeqsvv8uo1ixmyo, 2026-08-14 (journal timeline, zst.tmp artifacts with mtimes matching each killed attempt, one oom-kill at 08:31:54 +08:00).
Scope (one behavior: bounded, fault-tolerant recovery)
lob_archiver.rs:820 — stream the read (BufReader) instead of fs::read; no whole-file RAM spike.- Per-part fault tolerance: a part that fails recovery (or exceeds a size/time bound) is renamed
.quarantine, logged, and skipped — never fatal to process start.
zstd_timeout scaled by input size; a killed compression retries or quarantines, never aborts the process. Same treatment for oss_copy_timeout (binance-lob-archiver.rs:411).
Validation
Unit tests with synthetic oversized/corrupt parts: recovery completes, poisoned part quarantined, good parts recovered; memory bounded. Closest owning checks: cargo test -p hft-collector --locked.
Refs: V2 cutover incident 2026-08-14 (cutover evidence /data/monday/evidence/cutovers/20260814T004136Z-abeb35cc5f5e-908629/).
Problem
Production usdm wedged silently for ~36h (2026-08-12 13:04Z → 08-14 00:41Z) in a recovery crash loop: a 4.7GB
.partsegment could not be recovered byrecover_parts— itfs::reads the entire file into RAM (4.4G RSS peak,lob_archiver.rs:820), runs two full-file validation passes, then zstd with a fixed 300s timeout (lob_archiver.rs:687) that cannot finish on a pressured host →main()exits 1 →Restart=always→ identical failure forever. One oversized poisoned part permanently blocked the whole lane and later blocked the V2 cutover drain.Evidence: investigation on host i-6we6afeqsvv8uo1ixmyo, 2026-08-14 (journal timeline, zst.tmp artifacts with mtimes matching each killed attempt, one oom-kill at 08:31:54 +08:00).
Scope (one behavior: bounded, fault-tolerant recovery)
lob_archiver.rs:820— stream the read (BufReader) instead offs::read; no whole-file RAM spike..quarantine, logged, and skipped — never fatal to process start.zstd_timeoutscaled by input size; a killed compression retries or quarantines, never aborts the process. Same treatment foross_copy_timeout(binance-lob-archiver.rs:411).Validation
Unit tests with synthetic oversized/corrupt parts: recovery completes, poisoned part quarantined, good parts recovered; memory bounded. Closest owning checks:
cargo test -p hft-collector --locked.Refs: V2 cutover incident 2026-08-14 (cutover evidence
/data/monday/evidence/cutovers/20260814T004136Z-abeb35cc5f5e-908629/).