Skip to content

Harden research and live trading risk loop - #53

Closed
proerror77 wants to merge 7 commits into
mainfrom
codex/prediction-research-mission-loop
Closed

proerror77 wants to merge 7 commits into
mainfrom
codex/prediction-research-mission-loop

Conversation

@proerror77

Copy link
Copy Markdown
Owner

What changed

  • bind backtests to collector manifests, hashes, sequence coverage, replay checkpoints, and deterministic point-in-time materialization
  • model fees, L2 depth participation, slippage bands, partial entry/exit fills, and residual positions
  • reconcile authoritative balances, positions, open orders, and recent fills before live intake resumes
  • enforce conservative global, portfolio, and tokenized-security exposure across venues
  • export authoritative PnL, reconciliation truth, risk halts, and data-integrity gaps to health/readiness
  • require fresh authoritative reconciliation evidence before Agentic Paper/Shadow promotion

Why

The previous pipeline could accept self-attested backtest provenance, under-model executable liquidity, net exposure across unrelated venues, and treat incomplete reconciliation as healthy runtime evidence.

Impact

Research results now fail closed on unverified data. Live intake and Agentic promotion remain blocked whenever reconciliation, risk, or data integrity is incomplete. LiveSmall remains disabled.

Validation

  • 347 core/runtime/risk/domain tests passed
  • 35 Alpha Harness application tests passed
  • cargo check --workspace --all-targets
  • default backtest fixture replay
  • independent data/backtest and runtime/risk reviews: no P0/P1 findings

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 52 files, which is 2 over the limit of 50.

To get a review, narrow the scope:
• coderabbit review --type committed # exclude uncommitted changes
• coderabbit review --dir # limit to a subdirectory
• coderabbit review --base # compare against a closer base

Upgrade to a paid plan to raise the limit.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 43b0d38f-fdd2-4b55-9d06-dc9573800d13

📥 Commits

Reviewing files that changed from the base of the PR and between cfeb6d0 and 256b9b4.

⛔ Files ignored due to path filters (1)
  • rust_hft/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (52)
  • .gitignore
  • deploy/Dockerfile.hft
  • rust_hft/Cargo.toml
  • rust_hft/alpha-harness/app/src/loop_control.rs
  • rust_hft/alpha-harness/domain/src/lib.rs
  • rust_hft/apps/backtest/Cargo.toml
  • rust_hft/apps/backtest/src/config.rs
  • rust_hft/apps/backtest/src/engine.rs
  • rust_hft/apps/backtest/src/event.rs
  • rust_hft/apps/backtest/src/main.rs
  • rust_hft/apps/live/src/helpers/metrics.rs
  • rust_hft/apps/live/src/helpers/sentinel.rs
  • rust_hft/apps/live/src/main.rs
  • rust_hft/apps/live/src/runtime_attribution.rs
  • rust_hft/apps/live/tests/deployment_artifacts.rs
  • rust_hft/apps/live/tests/deployment_envelope.rs
  • rust_hft/config/backtest/default.yaml
  • rust_hft/config/dev/binance_prediction_live.yaml.example
  • rust_hft/config/dev/binance_prediction_quotes_only.yaml.example
  • rust_hft/data-pipelines/adapters/adapter-binance-prediction/Cargo.toml
  • rust_hft/data-pipelines/adapters/adapter-binance-prediction/src/lib.rs
  • rust_hft/data/backtest/sample.manifest.json
  • rust_hft/data/backtest/sample.ndjson
  • rust_hft/data/backtest/sample.raw.manifest.json
  • rust_hft/data/backtest/sample.raw.ndjson
  • rust_hft/data/backtest/sample.raw.ndjson._SUCCESS
  • rust_hft/deployment/docker/Dockerfile.trading
  • rust_hft/docker/Dockerfile
  • rust_hft/execution-gateway/adapters/adapter-ondo-perps/src/lib.rs
  • rust_hft/execution-gateway/adapters/adapter-polymarket/src/lib.rs
  • rust_hft/infra-services/core/metrics/src/http_server.rs
  • rust_hft/infra-services/core/metrics/src/lib.rs
  • rust_hft/market-core/core/src/types.rs
  • rust_hft/market-core/engine/src/execution_control.rs
  • rust_hft/market-core/engine/src/execution_queues.rs
  • rust_hft/market-core/engine/src/lib.rs
  • rust_hft/market-core/engine/tests/tokenized_security_execution_e2e.rs
  • rust_hft/market-core/ports/src/traits.rs
  • rust_hft/market-core/runtime/Cargo.toml
  • rust_hft/market-core/runtime/src/exposure_projection.rs
  • rust_hft/market-core/runtime/src/ipc_handler.rs
  • rust_hft/market-core/runtime/src/lib.rs
  • rust_hft/market-core/runtime/src/portfolio_manager.rs
  • rust_hft/market-core/runtime/src/risk_manager_factory.rs
  • rust_hft/market-core/runtime/src/system_builder.rs
  • rust_hft/market-core/runtime/src/system_builder/config_loader.rs
  • rust_hft/market-core/runtime/src/system_builder/venue_registry.rs
  • rust_hft/risk-control/portfolio-core/src/lib.rs
  • rust_hft/risk-control/risk/src/default_risk_manager.rs
  • rust_hft/risk-control/risk/src/sentinel.rs
  • rust_hft/tools/collector/src/bin/lob-pit-materializer.rs
  • rust_hft/tools/collector/src/lob_archiver.rs

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Review on demand using usage pricing
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/prediction-research-mission-loop

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@proerror77

Copy link
Copy Markdown
Owner Author

Agent Teams read-only review found merge blockers; keeping this PR draft and not merging:

  • P1 reconciliation coverage: reconcile(false) can publish complete/healthy authoritative account truth even though balances/positions/fills were not fully reconciled (execution_control.rs, invoked from unknown-order cancel in ipc_handler.rs). Only full reconciliation may update that truth, or coverage must be explicit.
  • P1 liveness/readiness coupling: /health returns 503 for readiness failures while Kubernetes uses it as liveness. Normal risk pause/reconciliation/data-gap states can trigger restart loops. Keep liveness process-only and strict gates on /readiness.
  • P1 real collector contract mismatch: the backtest adapter expects flat snapshot/l2_update/trade, but the Rust collector emits nested diff.frame / snapshot.snapshot and does not emit the fixture's fabricated trades. Manifest validation also does not bind market/symbol/dataset. Reuse/extend the governed materializer contract.
  • P2 schema fail-open: accept exactly the supported backtest-pit-v1, not any non-empty schema version.

Branch is also behind current main and has a .gitignore rebase conflict. Please split the live reconciliation/health changes from the backtest data-contract work, fix the above, rebase, and request re-review.

.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos();
let directory = std::env::temp_dir().join(format!(
.unwrap()
.as_nanos();
let directory =
std::env::temp_dir().join(format!("monday-backtest-chain-{}-{id}", std::process::id()));
@proerror77
proerror77 marked this pull request as ready for review July 16, 2026 03:47
@proerror77

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 256b9b4f6f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

let old_quantity = self.keyed_quantity.get(&key).copied().unwrap_or_default();
let next_quantity = old_quantity + incoming;
let old_notional = self.keyed_notional.get(&key).copied().unwrap_or_default();
let next_notional = next_quantity.abs() * price;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve prior order notional when projecting batches

When multiple same-symbol intents for the same venue/product are reviewed in one batch at different limit prices, this replaces the prior keyed notional with next_quantity * current_price. For example, a 10-lot buy at 100 followed by a 10-lot buy at 1 changes the projected gross notional from 1000 to 20, so ProjectedExposureRiskManager, portfolio budgets, and tokenized-security caps can approve a batch that actually commits far more notional than the configured limit.

Useful? React with 👍 / 👎.

.evidence_source
.as_deref()
.is_some_and(|source| !source.trim().is_empty())
&& context.evidence_venue == intent.target_venue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require a venue in tokenized evidence

For tokenized-security intents where both intent.target_venue and compliance_context.evidence_venue are unset, this equality treats missing venue evidence as valid. In the review() path, or in review_with_venue_specs() when the default venue fallback is used, such an intent can pass the securities-token filter with no venue-scoped eligibility/depth evidence, even though the policy is intended to require authoritative evidence for the execution venue.

Useful? React with 👍 / 👎.

Comment on lines 105 to 106
base_risk_manager,
system_risk_config.strategy_overrides.clone(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep projected exposure outside strategy overrides

When any strategy_overrides entry exists, this wraps the projected exposure manager inside StrategyAwareRiskManager. The engine calls review_with_venue_specs, but StrategyAwareRiskManager does not override that method, so the trait default calls review() separately per intent; the projected exposure layer only sees one order at a time, allowing cross-venue batches such as buy 60 on Binance and sell 60 on Bitget to bypass a 100-unit gross cap whenever an override is configured.

Useful? React with 👍 / 👎.

Comment on lines +320 to +321
&& context.evidence_observed_at.is_some_and(|observed_at| {
hft_core::now_micros().saturating_sub(observed_at) <= self.evidence_max_age_us

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject future-dated compliance evidence

When evidence_observed_at is greater than the local clock, saturating_sub returns 0, so future-dated tokenized-security evidence is treated as fresh. With a clock-skewed or malformed evidence producer, stale market-quality/corporate-action evidence can remain valid until wall clock catches up; require observed_at <= now before applying the max-age check.

Useful? React with 👍 / 👎.

Comment on lines +800 to +801
let reconciliation_ready = match truth_age_secs {
None => false,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep quotes-only pods ready without reconciliation

The production Kubernetes baseline starts from quotes_only: true and probes /readiness, but quotes-only runtimes never publish RuntimeTruthStatus because no live balance reconciliation runs. This new None => false path makes /readiness return 503 forever even after the market-data runtime is healthy, so the baseline pod never becomes Ready.

Useful? React with 👍 / 👎.

Comment on lines +629 to +632
&& (!runtime_truth.reconciliation_complete
|| !runtime_truth.reconciliation_healthy
|| runtime_truth.observed_at_us == 0
|| reconciliation_age_us > MAX_RECONCILIATION_AGE_US)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Produce reconciliation truth for paper/shadow activations

For Paper and Shadow deployments, the activation adapter sets the venue to execution_mode = Paper, while SystemRuntime only publishes reconciliation truth for live-mode balance reconciliation. With this new condition, every otherwise healthy PortfolioSnapshot in Paper/Shadow has observed_at_us == 0 and is rewritten to Decayed, so runtime_stage_is_healthy can never accept Paper/Shadow evidence for promotion.

Useful? React with 👍 / 👎.

@proerror77
proerror77 marked this pull request as draft July 16, 2026 04:56
@proerror77

Copy link
Copy Markdown
Owner Author

Final Agent Teams review found remaining merge blockers, so this PR is returned to Draft.

Blockers: post-submit duplicate OrderId can create an untracked cross-client order; Paper/Shadow promotion truth and quotes-only readiness are unreachable in the real startup path; projected exposure lacks pending-order reservations and can reject reduce-only exits; source-segment validation still has an in-place inode mutation gap.

Local integration fixes and tests are preserved, but will not be pushed as merge-ready until these contracts are closed and independently re-reviewed.

@proerror77

Copy link
Copy Markdown
Owner Author

Superseded by focused draft PRs: #59 (Prediction L2 market data), #60 (authoritative account reconciliation), #61 (batch exposure and tokenized policy), #62 (runtime health/readiness), #64 (LiveSmall promotion gate), and #65 (evidence-bound executable L2 replay).

The prior Paper/Shadow loop_control test-only reconciliation metrics were intentionally not carried forward: the replacement policy requires authoritative reconciliation only for LiveSmall, while Paper/Shadow remain simulated-execution stages. No replacement PR has been merged automatically.

@proerror77 proerror77 closed this Jul 16, 2026
@proerror77
proerror77 deleted the codex/prediction-research-mission-loop branch July 16, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants