Skip to content

fix(collector): accept closed same-hour reference fragments #175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
proerror77 merged 1 commit into from
Jul 20, 2026
Merged

fix(collector): accept closed same-hour reference fragments #175

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 60 additions & 3 deletions rust_hft/tools/collector/src/polymarket_research_import.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -649,8 +649,9 @@ pub(crate) fn with_validated_research_segments<T>(
.context("reference archive byte total overflow")?;
if let Some(previous) = references.last() {
let previous = segment_hour(&previous.identity)?;
if segment_hour(&reference.identity)? != previous + TimeDelta::hours(1) {
bail!("reference segments must be consecutive UTC hours");
let current = segment_hour(&reference.identity)?;
if current < previous || current > previous + TimeDelta::hours(1) {
bail!("reference segments must be same or consecutive UTC hours");
Comment on lines +653 to +654

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject backward same-hour fragments

When two fragments are in the same UTC hour, this condition accepts either order because current == previous regardless of their start_recorded_at/end_recorded_at values. A fragment recorded at 05:30 followed by one recorded at 05:10 therefore passes validation, after which combine_references concatenates them in that caller-supplied order and downstream selection retains the first metadata occurrence, making evidence provenance and output depend on an invalid ordering. Compare the authenticated timestamps for equal-hour fragments so backward inputs remain fail-closed.

Useful? React with 👍 / 👎.

Comment on lines +653 to +654

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject duplicate reference fragment identities

When an identical same-hour triplet is supplied more than once and that fragment contains no trade rows (for example, a metadata/settlement-only fragment alongside a separate trade fragment), this relaxed check accepts it and the later duplicate guard cannot detect it because it only tracks polymarket_trade record IDs. Validation then reports the same immutable source multiple times and concatenates its records multiple times, so a caller mistake changes authenticated provenance despite adding no new source data. Track and reject duplicate segment digests or file identities before combining the fragments.

Useful? React with 👍 / 👎.

}
}
references.push(reference);
Expand Down Expand Up @@ -970,6 +971,62 @@ mod tests {
rejects(&config, "consecutive UTC hours");
}

#[test]
fn validates_closed_reference_fragments_from_the_same_hour() {
let temp = tempfile::tempdir().unwrap();
let root = fs::canonicalize(temp.path()).unwrap();
let market = triplet(&root, "crypto_expiry", &market_rows());
let first = triplet(
&root,
"crypto_expiry_reference",
&[row(0, metadata("market_metadata")), row(1, trade())],
);
let second = triplet(
&root,
"crypto_expiry_reference",
&[
row(0, metadata("market_metadata")),
row(1, metadata("market_settlement")),
],
);
let config = ResearchSegmentValidationConfig {
market,
references: vec![first, second],
};

let report = validate_research_segments(&config).unwrap();

assert_eq!(report.references.len(), 2);
assert_eq!(report.references[0].hour, "05");
assert_eq!(report.references[1].hour, "05");
}

#[test]
fn rejects_skipped_reference_hours() {
let temp = tempfile::tempdir().unwrap();
let root = fs::canonicalize(temp.path()).unwrap();
let market = triplet(&root, "crypto_expiry", &market_rows());
let first = triplet(
&root,
"crypto_expiry_reference",
&[row(0, metadata("market_metadata")), row(1, trade())],
);
let mut second = vec![
row(0, metadata("market_metadata")),
row(1, metadata("market_settlement")),
];
for record in &mut second {
record["recorded_at"] = json!("2026-07-17T07:01:00Z");
}
let second = triplet(&root, "crypto_expiry_reference", &second);
let config = ResearchSegmentValidationConfig {
market,
references: vec![first, second],
};

rejects(&config, "same or consecutive UTC hours");
}

#[test]
fn accepts_reference_segments_with_event_local_trade_completion_proof() {
let mut reference = reference_rows(true);
Expand Down Expand Up @@ -1029,7 +1086,7 @@ mod tests {
row(2, metadata("market_settlement")),
];
for record in &mut second {
record["recorded_at"] = json!("2026-07-17T06:01:00Z");
record["recorded_at"] = json!("2026-07-17T05:02:00Z");
}
let second = triplet(&root, "crypto_expiry_reference", &second);
let config = ResearchSegmentValidationConfig {
Expand Down
Loading