fix(collector): bind shadow parity to event time

[proerror77]

Outcome

Fixes the failed one-hour Rust Polymarket shadow gate without weakening the fail-closed migration boundary.

Root cause

The first production-eligible shadow was stable for the full hour with identical trade IDs/counts and no duplicates, but the verifier compared settlements by collector retrieval time. Independent 30-second polling placed the same closed markets on opposite sides of the UTC boundary. It also required exact metadata set equality even when Rust validly discovered the next markets first.

Changes

• Compare mature settlements by market end time with a 15-minute lookback and 10-minute maturity lag.
• Clamp the event window to the Rust shadow start and bind derived window metrics in policy.
• Require legacy metadata/settlements to be subsets of Rust, while permitting only validated Rust additions.
• Reuse the canonical upload validators for active metadata, trades, and settlements.
• Bind every trade/settlement to its market metadata and validate each in-window record, not a historical field union.

Verification

• Rust parity tests: 10 passed
• Polymarket raw-ops control-plane tests: passed
• hft-collector library Clippy with -D warnings: passed
• rustfmt, ShellCheck, git diff check: passed
• independent review: no findings
• 3,600 UTC start-offset simulation: no pre-shadow lookback and at least 300 seconds parity tail

A new full one-hour cloud shadow is still required after merge; no cutover is authorized by the failed prior evidence.

Summary by CodeRabbit

• New Features

  • Strengthened shadow-parity validation for trades, market metadata, and settlement records.
  • Added settlement maturity and lookback safeguards to ensure comparisons use valid event windows.
  • Improved detection of missing records, mismatched values, invalid timestamps, and inconsistent market context.

• Documentation

  • Documented legacy-versus-Rust comparison rules, including record matching, settlement parity, required fields, and permitted extra records.

• Tests

  • Expanded coverage for settlement-window clamping, metadata and settlement mismatches, invalid windows, and late-arriving trades.

[coderabbitai]

Warning

Review limit reached

@proerror77, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 41 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 40a6fec3-a65e-460b-a5e0-5f8fcda2c1e9

📥 Commits

Reviewing files that changed from the base of the PR and between 428d677 and 0518229.

📒 Files selected for processing (6)

• deployment/aliyun/README.md
• deployment/aliyun/polymarket-raw-ops-shadow-gate.sh
• deployment/aliyun/polymarket-shadow-gate-policy.jq
• deployment/aliyun/test-polymarket-raw-ops-control-plane.sh
• rust_hft/tools/collector/src/polymarket_parity.rs
• rust_hft/tools/collector/src/polymarket_upload.rs

📝 Walkthrough

Walkthrough

The PR tightens Polymarket shadow parity around governed metadata, trades, settlements, and settlement-safe time windows. Rust comparison logic, shared validators, deployment gate policy, fixtures, regression tests, and parity documentation are updated together.

Changes

Polymarket shadow parity

Layer / File(s)	Summary
Canonical validation helpers rust_hft/tools/collector/src/polymarket_upload.rs	Shared trade, metadata, and settlement validators are exposed within the crate, and tape scanning delegates metadata validation to the shared helper.
Governed parity comparison rust_hft/tools/collector/src/polymarket_parity.rs	Metadata, trade, and settlement ingestion now validates normalized governed contracts, filters settlements by market maturity, compares context and identity bindings, and emits expanded evidence metrics.
Settlement-safe gate enforcement deployment/aliyun/polymarket-raw-ops-shadow-gate.sh, deployment/aliyun/polymarket-shadow-gate-policy.jq, deployment/aliyun/README.md	Gate execution clamps and validates parity windows, while policy checks governed shared values, record coverage, settlement bounds, and metadata context. Documentation records the updated parity rules.
Regression fixtures and gate coverage deployment/aliyun/test-polymarket-raw-ops-control-plane.sh, rust_hft/tools/collector/src/polymarket_parity.rs	Fixtures and assertions cover derived identities, mature settlements, metadata and settlement omissions, mismatches, malformed windows, duplicate trades, and invalid active trades.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• proerror77/monday#34: Earlier Rust Polymarket shadow/parity flow and raw-ops verification changes.
• proerror77/monday#44: Shadow-gate and watchdog gate process updated by this PR’s settlement timing checks.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main change: shadow parity is now bound to event time rather than raw timestamps.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/polymarket-parity-window-fix

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa2e2d9ffd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[proerror77]

@codex review

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust_hft/tools/collector/src/polymarket_parity.rs`:
- Around line 532-564: Keep the existing retrieved_at presence and timestamp
validation in the settlement normalization flow, but remove retrieved_at from
the normalized settlement object returned by the relevant projection so
shared_values_match does not compare collector-specific polling times. Apply the
same exclusion to the corresponding settlement projection at the additionally
referenced location, while preserving all immutable settlement fields.
- Around line 814-824: The parity evidence currently records settlement counts
and IDs without content-addressing the normalized settlement data. In the
settlement comparison flow around the evidence construction, add a deterministic
digest of the normalized settlement set and include that digest in the emitted
metrics/evidence alongside the existing settlement fields, reusing the
established normalization and digest helpers used for trades and metadata.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 428dfcde-1b6f-4263-867c-51666300ac6a

📥 Commits

Reviewing files that changed from the base of the PR and between d682082 and 428d677.

📒 Files selected for processing (6)

• deployment/aliyun/README.md
• deployment/aliyun/polymarket-raw-ops-shadow-gate.sh
• deployment/aliyun/polymarket-shadow-gate-policy.jq
• deployment/aliyun/test-polymarket-raw-ops-control-plane.sh
• rust_hft/tools/collector/src/polymarket_parity.rs
• rust_hft/tools/collector/src/polymarket_upload.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 428d677ec0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".