fix(data): enforce Binance raw trade range continuity - #122
proerror77 wants to merge 2 commits into
Conversation
|
Warning Review limit reached
Next review available in: 53 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Closing after focused protocol review. Binance documents f/l as the first and last raw trade IDs inside one aggTrade event, but does not guarantee cross-event previous.l + 1 == next.f; USD-M also excludes insurance-fund and ADL trades from aggTrade aggregation. Treating this undocumented relation as a shared Spot+USD-M trust invariant can reject legitimate data. Aggregate-trade ID continuity remains the supported frame-loss guard. This PR also picked up an unrelated collector clippy fix already owned by merged #118, so it is not a valid focused rollback unit. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e036584d3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let expected_first_trade_id = previous_last_trade_id | ||
| .checked_add(1) | ||
| .context("aggregate trade raw trade id overflow")?; | ||
| if trade.first_trade_id != expected_first_trade_id { |
There was a problem hiding this comment.
When MARKET=usdm, Config::stream_urls subscribes to Binance's futures @aggTrade stream, whose docs say only market trades are aggregated and insurance-fund/ADL trades are not returned (https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/All-Market-Mini-Tickers-Stream#aggregate-trade-streams). If one of those omitted executions consumes a raw trade ID, the next valid aggregate trade can have f > previous_l + 1; this equality then writes aggregate_trade_gap, returns an error, and the archiver keeps reconnecting from the stale validator state rather than accepting the live futures stream.
Useful? React with 👍 / 👎.
Change contract: reject Binance aggregate-trade sequences when the raw trade id range is not contiguous between consecutive events.
Out of scope: source-clock tolerance, replay-unsafe recovery, downstream assembler changes, and any new schema fields.
Dependency or merge order: independent of #121; may merge after or before it, but does not depend on additional Binance changes.
Focused validation: cargo check -p hft-collector --locked; cargo clippy -p hft-collector --all-targets --features collector-binance --no-deps --locked -- -D warnings; cargo test -p hft-collector --features collector-binance --locked.
Rollout/rollback impact: narrow validator tightening only; rollback is the reverse commit with no data migration.