feat(collector): materialize externally verified Binance PIT rows - #132
Conversation
|
Warning Review limit reached
Next review available in: 27 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| fn new(rows: &[Value]) -> Self { | ||
| let id = NEXT_FIXTURE_ID.fetch_add(1, Ordering::Relaxed); | ||
| let directory = std::env::temp_dir() | ||
| let requested_directory = std::env::temp_dir() |
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bdd48c73f2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| { | ||
| bail!("checkpoint is not replay safe"); | ||
| let triplet = BinanceMarketTapeTriplet { | ||
| data: path.clone(), |
There was a problem hiding this comment.
When --segment is provided as a relative path, this now passes that relative PathBuf directly into seal_binance_market_tape_triplet; I checked validate_triplet_paths, and it rejects any triplet path that is not absolute and normalized. The previous materializer accepted relative segment paths, and the documented/repro style invokes --segment part-...jsonl.zst, so this migration will fail before verifying otherwise-valid local segments unless the CLI canonicalizes the data path first and derives the sibling paths from that canonical path.
Useful? React with 👍 / 👎.
Change contract: lob-pit-materializer accepts Binance market-tape v1 segments only with externally supplied content and manifest SHA-256 anchors, verifies the complete segment set through VerifiedBinanceMarketTape, and emits the existing binance-lob-pit-v1 output after applying all equal-timestamp book events before one sample.
Out of scope: Collector startup or upload behavior, production service changes, evaluator policy changes, Polymarket snapshot compilation, deployment, and live trading.
Dependency or merge order: PR #131 is already merged and supplies the typed chronological replay events. This PR targets main and can merge independently.
Focused validation: GitHub/Linux run 29640828416 passed rustfmt on the target source, cargo test -p hft-collector --bin lob-pit-materializer --features collector-binance --locked, and the matching no-deps clippy lane with warnings denied. Targeted counterexamples cover mismatched anchor vector lengths before file access, a tampered segment retaining the original external anchor, and multiple updates at one timestamp being applied before exactly one sample.
Rollout/rollback impact: Offline research materialization only; no production collector or execution unit changes. Roll out by publishing a research image at the merged SHA and invoking the binary with controller-frozen digests. Roll back by reverting this PR or pinning the prior research image; existing immutable PIT outputs remain unchanged.