fix(runtime): keep Emergency sticky against degrade - #1153
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8fa98f87d0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if self.state == SentinelState::Emergency { | ||
| return SentinelAction::Continue; | ||
| } |
There was a problem hiding this comment.
当首次 emergency_stop(true) 因控制通道超时、适配器错误或部分撤单失败而返回错误或不完整报告时，这个提前返回会令后续检查永远只产生 Continue；因此 run_sentinel_loop 不会再次调用紧急撤单，而 OMS 中仍为开放状态的订单可能继续留在交易所。引擎模式保持 Emergency 并不能撤掉这些既有订单；应在锁定 Sentinel 状态的同时保留有界的 EmergencyExit 撤单重试路径。
AGENTS.md reference: AGENTS.md:L103-L104
Useful? React with 👍 / 👎.
673e9e4 to
c9a0d0b
Compare
enter_degrade_mode, Sentinel recovery, and gRPC degrade no longer downgrade Emergency to Degraded, so new order intents stay rejected until restart. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Owning-package Clippy compiles the included proto trait, and tonic::Status trips result_large_err. Keep Emergency degrade fail-closed without that unrelated lint failing the crate. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Keep Sentinel Emergency sticky against Degrade, but emit a bounded EmergencyExit path so an incomplete or failed emergency_stop can still retry venue cancels. A complete cancel stops further retries. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Monorepo CI failed on hft-collector concurrent legacy-tape parity (append rate below 100 rows/s). That test is outside the Emergency sticky diff; retrigger without changing collector. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
1cee072 to
7b860b8
Compare
Change
Emergency mode stays latched until process restart.
enter_degrade_mode, Sentinel recovery/force-recover, and the gRPC degrade RPC no longer rewrite Emergency to Degraded, soensure_accepting_new_intentscontinues to reject new orders.After the first Emergency latch, Sentinel still emits a bounded number of
EmergencyExitactions so an incomplete or failedemergency_stop(true)can retry venue cancels. A complete cancel stops further retries. Engine Emergency still does not cancel those leftover orders by itself.Issue relationship
None
Validation
Smallest disproof: after
emergency_exit(),enter_degrade_mode/ Sentinel degrade-bandcheck/ gRPCenter_degrade_modemust leaveTradingMode::Emergencyand keepsubmit_order_intentrejected.Cancel-retry disproof: after the first
EmergencyExit, degrade-bandcheckmust keep returningEmergencyExituntil the retry budget is spent ormark_emergency_cancel_completeruns, and must never returnDegrade.Owning checks:
cargo test -p hft-engine --locked emergency_mode_cannot_be_downgraded,cargo test -p hft-risk --locked emergency_,cargo test -p hft-grpc-server --locked enter_degrade_mode_cannot_leave_emergency,cargo test -p hft-live --locked sentinel_degrade.Runtime impact and rollback
Fail-closed control-plane change only. No live enablement, envelope activation, or collector/research path. Rollback is revert of this commit. Merge may awaken automatic GHCR publication of the merged SHA; that is not deployment authority.