fix(collector): skip empty reference batches

[proerror77]

Change contract

Skip only exactly empty USD-M reference batch directories so one abandoned empty directory cannot block later complete batches. Partial and malformed batches remain fail-closed and retained.

Issue relationship

None

Out of scope

Production mutation, collector cutover, reference schema changes, and PIT research contract changes.

Dependencies and merge order

None

Focused validation

• cargo test -p hft-collector --locked
• cargo clippy -p hft-collector --locked --lib --bin binance-usdm-reference-upload -- -D warnings
• git diff --check
• Counterexamples: an exactly empty batch is removed and later complete batches upload; a partial triplet is retained and fails before OSS access.

Rollout and rollback

No runtime mutation in this PR. Deploy only after merge from an exact-main release; rollback restores the prior digest-addressed uploader.

Scope exception

None

[coderabbitai]

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 28 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 31f0dcea-6ad1-4ed2-90c4-52f8defc3c83

📥 Commits

Reviewing files that changed from the base of the PR and between 52380a7 and cb3f6d2.

📒 Files selected for processing (2)

• agent-worktree.yml
• rust_hft/tools/collector/src/binance_usdm_reference_upload.rs

📝 Walkthrough

Walkthrough

The worktree contract now targets USDM reference empty-batch recovery. The Binance USD-M uploader removes empty batch directories, skips them during discovery, and preserves fail-closed handling for partial batches.

Changes

USDM empty-batch recovery

Layer / File(s)	Summary
Worktree contract scope agent-worktree.yml	The metadata now uses the USDM reference empty-batch recovery contract and scopes the collector file.
Batch discovery and validation rust_hft/tools/collector/src/binance_usdm_reference_upload.rs	reference_batch returns Option<ReferenceBatch>. Empty directories are removed and skipped. Non-empty directories still require the exact data, manifest, and _SUCCESS files. Tests cover successful uploads and partial-batch failures.

Estimated code review effort: 2 (Simple) | ~15 minutes

Possibly related PRs

• proerror77/monday#254: Both changes modify Binance USD-M reference-batch handling.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: skipping empty reference batches.
Description check	✅ Passed	The description includes all required sections and clearly states the behavior, safety boundaries, validation, rollout, and rollback plan.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/usdm-reference-empty-batch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@agent-worktree.yml`:
- Line 9: Update the no_external_runtime_mutation configuration key in
agent-worktree.yml from false to true, preserving the rest of the configuration
unchanged.

In `@rust_hft/tools/collector/src/binance_usdm_reference_upload.rs`:
- Around line 739-744: Update the test around publish_batch so the empty
canonical batch at RECEIVED_NS + 100 is created before publishing the complete
batch at RECEIVED_NS + 200. Preserve the existing batch_dir and
directory-creation setup while swapping the timestamps to exercise an empty
batch preceding a later complete batch.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 74d781be-f09a-4054-96a4-6c8d11fe1649

📥 Commits

Reviewing files that changed from the base of the PR and between e2a66fc and 52380a7.

📒 Files selected for processing (2)

• agent-worktree.yml
• rust_hft/tools/collector/src/binance_usdm_reference_upload.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52380a7b19

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".