CEX: atomically publish USD-M reference artifacts - #254
Merged
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds a public Binance USD-M reference artifact module that serializes complete batches into partitioned NDJSON, manifest, and ChangesBinance USD-M reference artifacts
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant publish_reference_batch
participant ArtifactStorage
participant verify_reference_artifact
Caller->>publish_reference_batch: CompleteReferenceBatch and configuration
publish_reference_batch->>ArtifactStorage: Write staged NDJSON, manifest, and _SUCCESS
publish_reference_batch->>ArtifactStorage: Atomic no-clobber directory rename
ArtifactStorage-->>Caller: PublishedReferenceArtifact with digests
Caller->>verify_reference_artifact: Published artifact and expected digests
verify_reference_artifact->>ArtifactStorage: Read and validate artifact members
ArtifactStorage-->>verify_reference_artifact: File contents and metadata
verify_reference_artifact-->>Caller: Reconstructed CompleteReferenceBatch
Possibly related PRs
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Comment on lines
+326
to
+334
| if unsafe { | ||
| libc::renameat2( | ||
| libc::AT_FDCWD, | ||
| source.as_ptr(), | ||
| libc::AT_FDCWD, | ||
| target.as_ptr(), | ||
| libc::RENAME_NOREPLACE, | ||
| ) | ||
| } != 0 |
Comment on lines
+346
to
+354
| if unsafe { | ||
| libc::renameatx_np( | ||
| libc::AT_FDCWD, | ||
| source.as_ptr(), | ||
| libc::AT_FDCWD, | ||
| target.as_ptr(), | ||
| libc::RENAME_EXCL, | ||
| ) | ||
| } != 0 |
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change contract
Atomically publish one complete Binance USD-M reference batch as a canonical NDJSON/manifest/SHA/_SUCCESS directory and strictly read it back using external data and manifest SHA trust anchors.
Part of #222. This PR does not close the issue.
Out of scope
Dependency or merge order
Depends on merged PR #253. Merge this PR before the CLI/health and deployment-gate layers.
Focused validation
not implementedcargo test -p hft-collector binance_usdm_reference_artifact::tests— 5 passedcargo test -p hft-collector— 200 library + 39 archiver + 8 other tests passed; 2 existing tests ignoredcargo clippy -p hft-collector --all-targets -- -D warningsgit diff --checkCounterexamples cover stale batches, non-official sources, wrong external SHA anchors, tampered content, symlinked triplet members, and injected mid-publication failure with clean retry.
Rollout / rollback impact
Rollout: library-only artifact contract; no service or production host changes. Rollback: revert this PR without changing the already-merged source collector or any research/snapshot path.
Summary by CodeRabbit