feat(collector): add producer watchdog diagnostics

[proerror77]

Issue relationship

Closes #829

Summary

• retain producer identity (producer id, endpoint, subscribed streams) for each current websocket shard, rebuilding metadata on every session
• record receive, enqueue, processed, and queue-health timestamps without cloning payloads
• include per-producer frame/enqueue ages and queue state in the existing process-watchdog firing log
• preserve an explicit unknown processed-age sentinel until the first processed event

Verification

• cargo test -p hft-collector --bin binance-lob-archiver --locked — 91 passed (local macOS run used a temporary RUSAGE_SELF shim only; restored before commit)
• cargo check -p hft-collector --bin binance-lob-archiver --tests --target x86_64-unknown-linux-gnu --locked
• cargo clippy -p hft-collector --bin binance-lob-archiver --tests --target x86_64-unknown-linux-gnu --locked -- -D warnings
• git diff --check

No watchdog threshold, reconnect policy, event/schema, dependency, runtime, or deployment behavior changes.

[coderabbitai]

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 43 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2fafbb6b-7b53-41fc-9256-5aa951d788e7

📥 Commits

Reviewing files that changed from the base of the PR and between bf3324c and 5156a3c.

📒 Files selected for processing (1)

• rust_hft/tools/collector/src/bin/binance-lob-archiver.rs

📝 Walkthrough

Walkthrough

The Binance LOB archiver now tracks per-producer receive and enqueue timestamps, processing progress, and queue health. Watchdog stall logs include these diagnostics. Session, rotation-barrier, and WebSocket paths update the metrics, with focused tests for stale and silent producers and saturated queues.

Changes

Watchdog diagnostics

Layer / File(s)	Summary
Diagnostic state and watchdog reporting rust_hft/tools/collector/src/bin/binance-lob-archiver.rs	The watchdog now stores per-producer timestamps, processed-data age, queue capacity, remaining capacity, and saturation state. Stall logs include one diagnostic record per producer.
Session and rotation processing integration rust_hft/tools/collector/src/bin/binance-lob-archiver.rs	Session startup resets diagnostics and records queue capacity. Producer events, archive events, periodic health reports, and rotation-barrier handling update processing and queue metrics.
WebSocket producer and enqueue tracking rust_hft/tools/collector/src/bin/binance-lob-archiver.rs	WebSocket paths record producer receive timestamps, queue health, and successful enqueue timestamps for buffered, subscription-proof, and normal events.
Diagnostic behavior tests rust_hft/tools/collector/src/bin/binance-lob-archiver.rs	Tests cover producer age calculations, stale and silent producer visibility, queue saturation, and updated rotation-barrier calls.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• proerror77/monday#40: Extends the same process watchdog in the Binance archiver.
• proerror77/monday#346: Modifies Binance archiver queue processing and watchdog handling.
• proerror77/monday#446: Modifies Binance LOB WebSocket event handling and timestamp tracking.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 35.71% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description explains the change and validation, but it omits most required template sections, including change contract, scope, dependencies, rollout, and scope exception.	Complete the required template sections and state None where they do not apply; retain the issue relationship and focused validation details.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address the linked issue by adding bounded per-producer telemetry, stall diagnostics, queue state, and focused tests without changing watchdog behavior.
Out of Scope Changes check	✅ Passed	The reported changes are limited to producer watchdog diagnostics and related tests, with no unrelated scope identified.
Title check	✅ Passed	The title clearly summarizes the main change: adding producer watchdog diagnostics to the collector.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/lob-watchdog-producer-diagnostics

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust_hft/tools/collector/src/bin/binance-lob-archiver.rs`:
- Around line 2764-2766: Move QueueHealth::from_sender recording in the send
flow so it occurs only after successful SendOutcome::Sent, ensuring the watchdog
observes the post-enqueue queue state. Apply this in
rust_hft/tools/collector/src/bin/binance-lob-archiver.rs at lines 2764-2766,
2654-2656 for each buffered proof event, and 2674-2681 after the pre-proof
event; retain existing shutdown and other send-outcome handling.
- Around line 1029-1030: Update the reconnect/session setup around
ProducerDiagnostics::new, run_session, and reset_session_diagnostics to rebuild
producer metadata from each session’s current stream_shards before spawning
producers, rather than only clearing timestamps. Ensure watchdog diagnostics
reflect the current producer IDs, URLs, and stream groups after runtime symbol
exclusions, while preserving each active WebSocket shard’s endpoint and
subscribed streams. Add a reconnect test that changes the active symbol set and
verifies the rebuilt metadata.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 42a9ac0d-9b20-46ac-8c1f-5282a87a1b0c

📥 Commits

Reviewing files that changed from the base of the PR and between 466bf77 and bf3324c.

📒 Files selected for processing (1)

• rust_hft/tools/collector/src/bin/binance-lob-archiver.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bf3324c352

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".