Uh oh!
There was an error while loading. Please reload this page.
fix(seal): stop design/real-seal epoch desync - #116
Conversation
Design re-read the chain schedule inside emit_leaves at late tempo, so an epoch-boundary flip could relabel the set as E+1 and permanently skip E. Real-seal only tried the current epoch, leaving /weights/latest stuck on an old chain-scale bundle. Catch up skipped epochs, widen the late window, walk back sealable epochs in prod-real-seal, and add validator pressure-verify logs when the sealed epoch lags the chain.
Warning Review limit reached
Next review available in:25 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe PR adds sequential design-emission catch-up with pinned blocks, coordination pressure warnings, and backward real-epoch seal retries using chain tempo and bounded walk-back depth. ChangesEpoch recovery and sealing
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant emitter_tick
participant design_emit_plan
participant emit_leaves_at
participant ChainState
emitter_tick->>design_emit_plan: request emission plan
design_emit_plan-->>emitter_tick: return epoch and pin_block
emitter_tick->>emit_leaves_at: emit planned leaves
emit_leaves_at->>ChainState: resolve hash at pin_block
ChainState-->>emit_leaves_at: return pinned hash
sequenceDiagram
participant prod_real_seal
participant ChainStorage
participant SealEndpoint
prod_real_seal->>ChainStorage: read tempo
ChainStorage-->>prod_real_seal: return tempo
prod_real_seal->>SealEndpoint: attempt candidate epoch seal
SealEndpoint-->>prod_real_seal: return HTTP status
prod_real_seal->>SealEndpoint: retry prior epoch when needed
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/design-challenge/src/orchestrator.rs`:
- Around line 314-327: Replace the catch-up cursor usage around design_emit_plan
and the award_round update at emitted_epoch with a persisted or reconstructable
contiguous next_uncovered_epoch checkpoint. Initialize it from durable coverage
on restart, advance it only after the next contiguous historical epoch is
successfully emitted, and never move it forward for noncontiguous current-epoch
awards. Add tests covering high-epoch restart catch-up and concurrent
award/emitter activity.
In `@crates/validator/src/epoch_loop.rs`:
- Around line 745-783: Update
tick_pressure_verify_allows_match_when_seal_lags_chain to configure the sealed
fixture response with a metagraph_block value below the fake chain’s far-ahead
current_block, ensuring the block-lag validation path executes while preserving
the expected ComparisonOutcome::Match assertions.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 67794d20-f67c-4852-b6be-cb960bd1782a
📒 Files selected for processing (6)
crates/design-challenge/src/lib.rscrates/design-challenge/src/orchestrator.rscrates/validator-verify/src/coordination.rscrates/validator/src/epoch_loop.rsdeploy/AGENTS.mddeploy/scripts/prod-real-seal.sh
| let last = self.emitted_epoch.load(Ordering::Relaxed); | ||
| let Some(plan) = design_emit_plan( | ||
| last, | ||
| epoch, | ||
| state.blocks_since_last_step, | ||
| tempo, | ||
| state.last_epoch_block, | ||
| ) else { | ||
| return Ok(false); | ||
| } | ||
| self.emit_leaves().await?; | ||
| }; | ||
| // Pin epoch + block from this tick's schedule snapshot — do **not** | ||
| // re-read chain inside emit (end-of-epoch flip used to relabel the set | ||
| // as E+1 and permanently skip E, starving real-seal with D24 409s). | ||
| self.emit_leaves_at(plan.epoch, plan.pin_block).await?; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Persist a contiguous catch-up checkpoint.
emitted_epoch starts at 0 in Orchestrator::new, but line 314 uses it as the catch-up cursor. After a restart at a high chain epoch, the emitter starts at epoch 1 and needs one tick per historical epoch before it reaches the recent gap.
fetch_max also cannot represent contiguous coverage. If award_round emits the current epoch while catch-up is processing an older epoch, line 1148 advances the value to the current epoch. The next tick then skips every remaining missing epoch.
Persist or reconstruct a contiguous next_uncovered_epoch checkpoint. Do not advance that checkpoint when a noncontiguous current-epoch emission succeeds. Add restart and concurrent award/emitter tests.
Also applies to: 1148-1148
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/design-challenge/src/orchestrator.rs` around lines 314 - 327, Replace
the catch-up cursor usage around design_emit_plan and the award_round update at
emitted_epoch with a persisted or reconstructable contiguous
next_uncovered_epoch checkpoint. Initialize it from durable coverage on restart,
advance it only after the next contiguous historical epoch is successfully
emitted, and never move it forward for noncontiguous current-epoch awards. Add
tests covering high-epoch restart catch-up and concurrent award/emitter
activity.
| #[tokio::test] | ||
| async fn tick_pressure_verify_allows_match_when_seal_lags_chain() { | ||
| // Same metagraph as the sealed fixture, but chain epoch/tip far ahead — | ||
| // pressure-verify warns (validator_seal_lag) yet Match must still proceed. | ||
| let epoch = 77u64; | ||
| let (client, _chain, trust, merkle_root, _) = sealed_match_fixture(epoch).await; | ||
| let miner = [0xA1u8; 32]; | ||
| let chain = FakeChain::new(FakeChainConfig { | ||
| current_block: 10_000, | ||
| subnet_epoch_index: epoch + 11, | ||
| hotkeys: vec![miner.to_vec()], | ||
| owner_hotkey: miner.to_vec(), | ||
| commit_reveal_enabled: true, | ||
| last_epoch_block: 500, | ||
| ..FakeChainConfig::default() | ||
| }); | ||
| let dedupe = EpochSubmitDedupe::new(); | ||
| let submit = CoordinationSubmitConfig { | ||
| netuid: 1, | ||
| hotkey: vec![0xBBu8; 32], | ||
| version_key: 3, | ||
| epoch_length: 360, | ||
| }; | ||
| let out = coordination_compare_once(&client, &chain, &trust, Some(&submit), &dedupe) | ||
| .await | ||
| .expect("ok") | ||
| .expect("some"); | ||
| match out { | ||
| ComparisonOutcome::Match { | ||
| epoch: e, | ||
| merkle_root: root, | ||
| .. | ||
| } => { | ||
| assert_eq!(e, epoch); | ||
| assert_eq!(root, merkle_root); | ||
| } | ||
| other => panic!("expected Match despite seal lag, got {other:?}"), | ||
| } | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Cover the stale metagraph_block branch.
The fixture response omits metagraph_block, so this test verifies epoch lag only. Add a sealed metagraph_block value below the far-ahead tip so the new block-lag path executes while ComparisonOutcome::Match remains unchanged.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/validator/src/epoch_loop.rs` around lines 745 - 783, Update
tick_pressure_verify_allows_match_when_seal_lags_chain to configure the sealed
fixture response with a metagraph_block value below the fake chain’s far-ahead
current_block, ensuring the block-lag validation path executes while preserving
the expected ComparisonOutcome::Match assertions.
Keep pinned-epoch emit + catch-up scheduling while landing under the design-challenge 1500 LOC gate so CI can merge the seal sync fix.
Uh oh!
There was an error while loading. Please reload this page.
Summary
emit_leaves, so an epoch-boundary flip could relabel the leaf set asE+1and permanently skipE.base-real-sealonly tried the current epoch → perpetual D24409, while/v1/weights/lateststayed pinned on an old chain-scale seal (burn seals cannot outrank it). Observed: seal stuck at 24412 for ~11 epochs while challenges were on 24423.prod-real-seal.sh: walk back up to 16 sealable epochs; clearer logs.validator_seal_lag/validator_seal_metagraph_stalewhen sealed epoch or metagraph block lags the chain.Ops evidence
base-real-seal.log: 409 from 24413→24423 until walk-back script deployed.base-prodsealed epoch 24423 (metagraph_block8815687).design-challenge(+ validator) for the emitter + pressure-verify log fix.Test plan
cargo test -p design-challenge --lib -- emit_plancargo test -p validator --lib -- tick_pressure_verifySummary by CodeRabbit
New Features
Bug Fixes