feat(research): publish verified CEX ResearchSnapshot V2 - #786
Conversation
📝 WalkthroughWalkthroughThe LOB materializer now produces V3 snapshots and PIT rows. It validates and republishes authenticated fee, USD-M reference, and runtime-latency evidence. USD-M rows include funding, funding costs, and open interest. ChangesLOB PIT materializer
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related issues
Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant CLI
participant lob_pit_materializer
participant EvidenceArtifacts
participant PITRows
CLI->>lob_pit_materializer: provide materialization and evidence inputs
lob_pit_materializer->>EvidenceArtifacts: validate and republish fee, reference, and latency artifacts
EvidenceArtifacts-->>lob_pit_materializer: return validated evidence
lob_pit_materializer->>PITRows: materialize V3 rows with funding and open interest
PITRows-->>CLI: return snapshot and materialization report
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6e3465826
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67467e7667
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c5ab91804
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d6aecaf52b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
💡 Codex Reviewmonday/rust_hft/tools/collector/src/bin/lob-pit-materializer.rs Lines 791 to 792 in 8a861d6  Reject conflicting observations at the same reference clock
When two valid reference artifacts contain different funding or OI values with the same AGENTS.md reference: AGENTS.md:L67-L68 monday/rust_hft/apps/live/src/runtime_attribution.rs Lines 469 to 471 in 8a861d6 Record availability only after durable feedback publication
When the materialization cutoff falls between this timestamp and AGENTS.md reference: AGENTS.md:L67-L68 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Materialize verified Binance Spot/USD-M market tapes into the V3 outer materialization carrying CexReplaySnapshotV2: PIT instrument rules, account-bound maker/taker fees, independent funding and OI timelines, aggregate-trade flow, and signed runtime lifecycle latency evidence. Rebased onto origin/main be9e746, which already contains the signed arrival-cost evidence (#787), its verification (#788), fee-account binding (#789), and fee snapshot timers (#793). Refs #794.
- Delegate runtime latency verification to the hardened
alpha_domain::runtime_latency_evidence verifier instead of a drifted
collector-local copy: event IDs are deduplicated before filtering
(conflicting duplicates fail closed), every fill must carry an
authenticated instrument_market_{market} identity, and USD-M fails
closed until a derivatives execution path exists.
- Republish verified fee and reference evidence triplets into the
materialization artifact directory under content-addressed names so
the digests recorded in the snapshot resolve to immutable bytes.
- Restructure the test fixture per market: the happy path is Spot
(signed LiveSmall latency evidence exists only for Spot), a new test
proves USD-M materialization fails closed, and the aggregate-trade
assertions now match the (previous, current] bucket semantics.
af055b9 to
9806c34
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9806c349c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Publish the complete digest-anchored runtime feedback log and the trusted-key document into the artifact directory under their content digests, so independent readers can reverify every signed lifecycle event and no resolvable-evidence gap remains for the key document. - Charge USD-M funding settlements over each row's actual (current, future] holding interval, matching how the evaluator applies funding_bps to the held position; a settlement that precedes the holding interval is no longer charged and multi-bucket horizons collect every settlement.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
rust_hft/tools/collector/src/bin/lob-pit-materializer.rs (1)
1790-1844: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftAdd coverage for the USD-M end-to-end path and the funding-reschedule branches.
rejects_usdm_without_a_derivatives_execution_pathasserts that USD-M materialization fails.charges_funding_over_the_holding_intervalcallsmaterialize_rowsdirectly with a hand-builtResearchContextV2. As a result, no test exercisesbind_usdm_referenceoutput through a successful materialization. The following new behavior has no assertion:
CexDerivativesReferenceV2assembly, includingfundingandopen_interestCexPitSeriesEvidenceV2observation counts andmax_gap_ns.evaluation_funding_bps_per_bucketpopulation at lines 399-407.- The USD-M required-modality set at lines 420-421 and the
Funding/OpenInterestrow modalities.Two funding branches at lines 1174-1175 are also unasserted, and the PR objectives call both out:
- A later pre-settlement observation announces a different
next_funding_at, so the settlement is dropped.- A replacement observation arrives exactly at the settlement time, so the charge is preserved because the predicate is
available_at < scheduled.Add cases to
charges_funding_over_the_holding_intervalfor both branches, and add a successful USD-M materialization test once the runtime-latency fixture supports the derivatives execution path.Also applies to: 1875-1887
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust_hft/tools/collector/src/bin/lob-pit-materializer.rs` around lines 1790 - 1844, Add coverage for the USD-M end-to-end materialization path, using the runtime-latency fixture once it supports derivatives execution, and assert successful CexDerivativesReferenceV2 assembly, funding/open-interest evidence counts and max_gap_ns, evaluation_funding_bps_per_bucket, required modalities, and Funding/OpenInterest row modalities. Extend charges_funding_over_the_holding_interval to assert both rescheduling branches: drop a settlement when a later pre-settlement observation changes next_funding_at, and preserve the charge when a replacement observation arrives exactly at settlement time.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust_hft/tools/collector/src/bin/lob-pit-materializer.rs`:
- Around line 611-621: Update
rust_hft/tools/collector/src/bin/lob-pit-materializer.rs at lines 611-621 to set
CexArtifactTripletV2.success_sha256 to the SHA-256 hex digest of
success.as_bytes(); at lines 728-734, return the marker digest from
republish_evidence_triplet instead of artifact.data_sha256, and have
republish_evidence_triplet return the digest it computes from the read marker
bytes; at lines 906-914, apply the same marker-digest correction to the USD-M
reference triplet. Extend preserves_report_evidence_and_point_in_time_rows with
assertions that each published _SUCCESS file’s digest matches its recorded
success_sha256.
- Line 185: Update the V2 materialization reader branch to deserialize
CexReplaySnapshotV2, matching the BINANCE_LOB_PIT_MATERIALIZATION_SCHEMA_V3
handling in mission_runner.rs. Replace the stale CexReplaySnapshotV1 type while
preserving the existing V2 materialization flow.
- Around line 399-407: Update the funding bound calculation in the
derivatives_reference mapping to preserve the (current, future] interval
semantics of funding_cost_bps. Ensure evaluation_funding_bps_per_bucket is
either renamed to reflect the full holding interval or populated with a
per-bucket average by applying label_horizon_buckets consistently before
cex_snapshot_costs() compares the values.
---
Nitpick comments:
In `@rust_hft/tools/collector/src/bin/lob-pit-materializer.rs`:
- Around line 1790-1844: Add coverage for the USD-M end-to-end materialization
path, using the runtime-latency fixture once it supports derivatives execution,
and assert successful CexDerivativesReferenceV2 assembly, funding/open-interest
evidence counts and max_gap_ns, evaluation_funding_bps_per_bucket, required
modalities, and Funding/OpenInterest row modalities. Extend
charges_funding_over_the_holding_interval to assert both rescheduling branches:
drop a settlement when a later pre-settlement observation changes
next_funding_at, and preserve the charge when a replacement observation arrives
exactly at settlement time.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: d14cd6a0-99f7-4558-9709-5633df43deb0
⛔ Files ignored due to path filters (1)
rust_hft/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (3)
agent-worktree.ymlrust_hft/tools/collector/Cargo.tomlrust_hft/tools/collector/src/bin/lob-pit-materializer.rs
Change contract
Materialize Binance Spot and USD-M market tapes into the V3 outer artifact carrying a verified ResearchSnapshot V2: PIT tick/step/min-notional, authenticated maker/taker fees, independent funding and OI, signed real lifecycle latency/slippage, and aggregate-trade flow.
Issue relationship
Refs #794
Rebase baseline
Rebased onto
origin/mainbe9e746a, which already contains #787 (signed arrival-cost evidence), #788 (verification), #789 (fee-account binding), and #793 (fee snapshot timers). The previous draft history merged those prerequisite branches; the rebuilt history is two commits on top of main: the materializer itself, plus the review-closure fixes.Review thread disposition (all 17 resolved)
Fixed in the follow-up commit:
Resolved by delegating to the merged
alpha_domain::runtime_latency_evidenceverifier (the drifted collector-local copy is deleted):instrument_market_{market}identity, USD-M hard-fails;arrival_slippage_bps) instead of limit-price-clampedrealized_slippage_bps; fills without the metric are skipped as inapplicable;Resolved by main (verified in code, evidence posted per thread):
MAX_FEE_EVIDENCE_GAP_NS90s, test rejects_sparse_fee_evidence) — implemented in this PR;CexReplaySnapshotV2::validate.Out of scope
Trading runtime mutation, derivatives execution, order authority, credential provisioning, production unit enablement, and production data deletion.
Focused validation
cargo check -p hft-collector --locked --target x86_64-unknown-linux-gnu: clean.cargo clippy -p hft-collector --locked --tests --target x86_64-unknown-linux-gnu: clean.cargo test -p hft-collector --locked --bin lob-pit-materializer: 8/8 pass (run on macOS via a temporary uncommittedRUSAGE_THREADcfg shim, since the crate'slibc::RUSAGE_THREADuse is Linux-only; Linux CI remains the authoritative lane).cargo test -p hft-collector --lockedlib suite: 326 passed, 4 failed inbinance_fee_uploadtests — pre-existing macOS tempdir-canonicalization failures on code untouched by this PR (identical to origin/main); expected green on Linux CI.Size note
The diff adds 869 non-generated lines (mostly the materializer test module), above the 750 guideline. Requesting reviewer judgment on an atomic exception rather than splitting tests from the code they pin.
Rollout and rollback
No runtime mutation in this PR. Production activation remains fail-closed until authenticated fee credentials, resolvable evidence artifacts, and matching signed LiveSmall lifecycle evidence are available. Rollback is the squash commit.
Scope exception
None.
Summary by CodeRabbit
New Features
Bug Fixes