CEX: WebSocket L2 delta 时序与连续性 collector

[proerror77]

Parent

#219

What to build

为 CEX 盘口采集可回放的 snapshot+delta：source sequence、source time、received time、重连/补 snapshot 证据和 gap 计数。它为亚秒盘口演化提供独立数据面。

Acceptance criteria

• 每个流有 snapshot seed、单调 sequence delta 与显式重连边界。
• sequence gap、时间回退、无 snapshot 的 diff 会 fail closed。
• manifest 提供 symbol coverage、gap/reconnect、延迟和深度统计。
• shadow gate 与 canonical artifact 回读通过。

Blocked by

None - can start immediately

[proerror77]

This was generated by AI during triage.

PR #264 merged as 7c0a69efbc1536844b322113b409799e240a18ab and PR #265 merged as a2018be6d305f109a9decaf522f69ac7622ccd61. Main now contains the complete code-side #224 contract:

• binance.lob_continuity.v1 manifest summaries with symbol coverage, snapshot/diff/checkpoint evidence, sequence/source-time continuity, source/received latency, update IDs, and depth statistics
• sealed raw-row recomputation and explicit packaged CLI strict mode
• monday.rust_lob_shadow_gate.v3 canonical OSS round-trip evidence and fail-closed promotion policy
• stable capture-session/reconnect-boundary enforcement and zero-gap/zero-rollback checks
• focused counterexamples, local deployment tests, two independent read-only reviews, and automated review fixes

No ECS service was installed, started, stopped, restarted, replaced, or cut over. Issue #224 remains open because the final acceptance criterion still requires an actual isolated >=3600-second shadow run and canonical artifact readback under one explicitly named production host controller. CI is not runtime evidence.

CI note: the review-fix head passed all #224-specific checks on each cloud attempt. The shared Rust Workspace later failed only in unchanged test-trading-ecs-host-contract.sh on three reruns; the original PR head had a complete Linux green run, and the exact chain plus ten isolated trading-test runs passed locally. That unrelated test instability was documented on PR #265 and was not mixed into this collector rollback unit.

[proerror77]

Runtime gate follow-up (single write owner: Codex root; branch codex/cex-l2-subscription-coverage-224; base/head before edits f5affe7a28aea0ea4ec61dc6ce0657b7475ddbcf).

The exact-main candidate b0d041d12973ca3cd598cbd0573efe04f1f2a2fbac85c13676d7b93e5787ae8f failed closed after the full 2400-second settle window. Cloud Assistant invocation: t-jpn6rtd62biz7r4; immutable run: /data/monday/evidence/shadow-gates/b0d041d12973ca3cd598cbd0573efe04f1f2a2fbac85c13676d7b93e5787ae8f/a8f41c36ff3a29302631d895d5582c27d7ddf198d0de848c5964b8052b07e8f4/runs/20260723T190010Z-357180/run.json; no PASSED.sha256. Production remained on c11f7aabd50b96a04ce1a5123865e4146f53a8fdffe2b6923fab32039b5d15fc, both production units active/running, both shadow units inactive/dead.

Focused failing proof: Spot reached 1374/1375; only BNSOLSOL remained snapshot-only. Three consecutive sealed 10-minute manifests recorded a two-sided snapshot/checkpoint with diff_count=0 and no BNSOLSOL trade summary. Binance REST showed its last aggregate trade was before the post-exclusion capture session; a direct 60-second websocket probe also received no event. This is a genuinely static stream, not a sequence gap.

Change contract: before any snapshot request, verify every depth and aggTrade websocket shard by exact LIST_SUBSCRIPTIONS set equality; persist subscription-coverage evidence; allow a two-sided snapshot with verified stream coverage and zero later events to be continuity-complete without fabricating a diff or trade. Observed events retain all existing sequence/trade-gap fail-closed checks. Out of scope: research/snapshot/evaluator logic, reference/funding/OI collectors, and any production cutover before a fresh >=3600-second gate plus canonical readback.

[proerror77]

Collector cutover controller: Codex /root remains the sole ECS writer. Layer A is now PR #293 at head 47655cd, based on main 4201492. Scope is verifier/data-contract only; no production change or cutover between layers. Layer B producer/deployment remains uncommitted and owned by the same controller.

[proerror77]

Layer B is now stacked PR #295 at head 7988bdb, based on layer A PR #293 head 47655cd. Merge order is #293 then #295; rollback order is #295 then #293. Codex /root remains the sole ECS controller; no runtime change has occurred.

[proerror77]

Status correction for the final issue stack:

• Layer A: fix(collector): verify complete market tape coverage #293 verifier/data contract — merged.
• Layer B: fix(collector): prove exact Binance stream coverage #295 producer/runtime exact stream-coverage proof — current head c74b787; CI running.
• Layer C: fix(deploy): gate complete Binance market tape #296 deployment/shadow-gate/cutover contract — stacked after fix(collector): prove exact Binance stream coverage #295.

Merge order is A -> B -> C; rollback order is C -> B -> A. Root remains the sole ECS controller. No ECS production unit has been changed during this PR phase; cutover remains forbidden until #295 and #296 merge and a fresh full shadow gate passes.

[proerror77]

Production gate result for candidate cdea978281805e2cde403941e29b3e9247ed6d1d6011f5eb217fac3653d6e055:

• Cloud Assistant invocation: t-jpn6rts76tsm96o
• Result: failed closed; no cutover.
• Exact failure: Spot manifest part-1784843998927960167.jsonl.zst.manifest.json started at 21:59:58 UTC, lasted 15.1s, and contained checkpoint=2750, stream_coverage=1, diff=1, agg_trade=0.
• Root cause: elapsed start+600s rotation occurred immediately before the UTC hour partition; checkpoint seeding crossed 22:00 and triggered a second rotation, creating a technical no-trade segment.
• All later full Spot segments contained real agg_trade counts (84210, 69043, 57202, 49470, 51001; final tail 8946), with LOB gaps 0.
• Follow-up PR: fix(collector): align Binance segment rotation #301 aligns segment rotation to UTC boundaries. The agg_trade gate remains unchanged.

Production remained on c11f7aabd50b96a04ce1a5123865e4146f53a8fdffe2b6923fab32039b5d15fc. A new full gate is required after #301 merges and a new candidate is built.

[proerror77]

Production gate attempt failed closed; no cutover occurred.

• candidate: 87143fa0acb8d28b37150e4047351524348f018db7834f563a8d0a4a9052783a
• source: a4834f673178147417fe3dfa6c017d03d4a671bc
• bundle: 15de90e99d9f66aa8cc9029235c240b955d90caee2ec40e1681f52f02006f895
• invocation: t-jpn6ruf8ajelw5c
• run: 20260724T020651Z-380249 (requested_duration_seconds=3600, test_only=false)
• terminal failure at 2026-07-24T03:02:03Z: usdm health failed during observation
• root cause evidence: USD-M WebSocket reset at 03:01:50Z; the collector reconnected and re-established 12 shards / 1146 streams at 03:01:51Z, but the capture session changed and snapshot readiness temporarily reset, so the continuity gate correctly rejected the run.
• systemd NRestarts=0, LOB sequence_gaps=0, upload failures/warnings=0.
• no gate.json and no PASSED.sha256 were emitted.
• both production units remained active on old SHA c11f7aabd50b96a04ce1a5123865e4146f53a8fdffe2b6923fab32039b5d15fc.

A single controlled full-duration retry will use the same immutable candidate. A repeated identical failure will be treated as a collector reconnect-boundary defect rather than retried again.

[proerror77]

2026-07-24 replacement gate update (candidate ed386df55ed88077d285d077c5f5f9adda9237d2a482d334ef15f9d88edaefc3, source 9b02d2e1812a1a0eee1c28055c18ab7dce58836e):

• Run 20260724T043802Z-395363 failed closed after 49m30s; no cutover occurred.
• Exact fault at 2026-07-24T05:27:16.091665Z: STARUSDT depth previous-update gap.
• The collector re-established USD-M stream coverage one second later, but the capture session changed, so the one-hour continuity gate correctly rejected the run.
• Shadow cleanup completed (NRestarts=0, units inactive); production stayed on SHA c11f7aabd50b96a04ce1a5123865e4146f53a8fdffe2b6923fab32039b5d15fc.
• A fresh full-duration gate is required; this gap will not be waived or represented as complete data.

[proerror77]

Audit status (2026-08-01)

代码合约已由 #248–#366 全部合入,但关闭证据不足:四次正式 shadow gate(07-23 ×2、07-24 ×2)均 fail-closed,#366 要求的"重建候选并重跑 ≥3600s 正式 gate + canonical 回读"无任何执行记录;#537(gate 运行所属 runtime issue)仍 OPEN,且 #534 记录生产 usdm LOB archiver 自 07-28 起停摆。

下一步:在 #534 修复生产单元后,由 named controller(kimi,2026-07-31 授权)基于含 #366 的 main 重建候选,按 #537 的 Runtime control 跑正式 ≥3600s gate,取得 PASSED.sha256 + canonical 回读后,再经 /pm:issue-close 提交证据关闭本 issue。

Code contract is fully merged (#248–#366), but closure evidence is missing: all four formal shadow gates (07-23 ×2, 07-24 ×2) failed closed, and the post-#366 "rebuild candidate and rerun the formal ≥3600s gate with canonical readback" was never executed or recorded. Issue #537 (the runtime issue owning this gate run) is still OPEN, and #534 records the production usdm LOB archiver down since 2026-07-28. Next step: after #534 restores the production unit, the named controller (kimi, authorized 2026-07-31) rebuilds a candidate from main incl. #366, runs the formal ≥3600s gate per #537's Runtime control, and only then closes this issue via /pm:issue-close with PASSED.sha256 + canonical readback evidence.