fix(collector): verify LOB OSS uploads by readback before deleting sources

[proerror77]

Change contract

LOB archiver 上传闭环:binance-lob-archiver 的 OSS 上传从"ossutil 返回成功即写 cleanup marker 并删本地源"改为 fail-closed 闭环——上传 data/manifest/_SUCCESS 三件套后重新从 OSS 下载三件,逐件 sha256 校验(data 对 manifest 声明的 digest、manifest 对本地重算 digest、_SUCCESS 内容逐字节比对),全部通过才走原有 cleanup marker 流程删除本地源并 fsync 父目录。远端三件套已存在且字节一致按幂等 retried 处理(直接清理本地,不上传);远端存在但哈希不一致 fail-closed:保留本地、记录 last_error、绝不删除、绝不覆盖(上传改用 --ignore-existing 替代 --force)。

同时 LOB 上传器写入与 fee 上传器同 schema 的 upload-status.json(updated_at/pending_batches/uploaded_batches/retried_batches/failed_batches/failure_count/discovery_failed/last_error_at/last_error/last_success_at),健康检查由此可见积压;每次成功上传留收据 last_uploaded_object(对象 URI)+ last_uploaded_triplet(object_prefix、data/manifest/success sha256、data_bytes、RFC3339 uploaded_at)+ last_success_at。fee/reference 上传器已写 last_success_at,无需补。

Issue relationship

None

Out of scope

• fee/reference 上传器行为不变;仅将 polymarket_upload::ExclusiveTempDir 提为 pub 供 bin 复用。
• polymarket_upload.rs 的 libc::RUSAGE_THREAD 在 macOS 宿主不编译(fix(collector): reuse sealed market tape validation #772 引入的 pre-existing 问题,CI 为 ubuntu);不在本 PR 修。
• 运行时部署/切流不属本 PR。

Dependencies and merge order

None

Focused validation

• cargo clippy -p hft-collector --all-targets --features collector-binance --no-deps --locked --target x86_64-unknown-linux-gnu -- -D warnings:通过(宿主为 macOS,crate 因上述 pre-existing 原因不能本机编译 lib,故用 Linux target 做编译/lint 验证)。
• cargo test -p hft-collector --features collector-binance --locked --bin binance-lob-archiver:80 passed / 0 failed,含 8 个新 focused 测试:
  • verified_readback_removes_the_local_triplet_and_returns_a_receipt(读回校验通过才删,收据字段齐全)
  • corrupted_readback_retains_the_local_triplet(读回字节被腐坏→报错、本地三件全留、无 cleanup marker)
  • conflicting_remote_triplet_fails_closed_and_retains_the_local_triplet(远端冲突→0 次上传、远端字节未被覆盖、本地保留)
  • matching_remote_triplet_is_an_idempotent_retry(远端一致→幂等 retried、0 次上传、本地清理)
  • upload_pending_reports_failures_per_segment_and_keeps_uploading(单段失败不阻塞其他段,pending_batches 可见)
  • upload_status_records_the_upload_receipt / upload_status_records_failures_and_keeps_the_backlog_visible / upload_status_records_a_discovery_failure(status json 全字段,failure_count 累计语义与 monday-collector-health.sh 一致)
• 全量 cargo test -p hft-collector:326 passed;4 个 binance_fee_upload 失败已在未改动基线(be9e746)上原样复现(macOS /var symlink 触发 canonical 检查,pre-existing,CI ubuntu 不受影响)。
• git diff --check:通过。
• 反例边界:冲突与腐坏用例证明删除门不可绕过——只有三件套读回哈希全对才会写 cleanup marker。

Rollout and rollback

影响 binance-lob-archiver 上传循环与 --upload-only 排水:上传失败/校验失败时段保留在 spool 重试,upload-status.json 出现 last_error 会使现有健康检查告警(语义不变,只是现在失败更可见)。旧版 UploadStatus 文件读取向后兼容(新字段 serde default)。回滚 = revert 本 PR 即恢复旧上传路径;无状态迁移。

Scope exception

701 新增行中约 420 行为 focused 测试与 fixture;实现本体约 230 行。读回校验、删除门控与 status 收据是同一个"上传闭环"行为,无法安全拆分(先上校验不带删除门=无闭环;先删本地不带校验=现状)。

[coderabbitai]

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4b5aa672-5b09-43ce-9be1-bc88709bb2a2

📥 Commits

Reviewing files that changed from the base of the PR and between be9e746 and 38ce590.

📒 Files selected for processing (3)

• rust_hft/tools/collector/src/bin/binance-lob-archiver.rs
• rust_hft/tools/collector/src/lob_archiver.rs
• rust_hft/tools/collector/src/polymarket_upload.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}