fix(dash-spv): reprocess blocks on rescan against newly derived scripts by xdustinface · Pull Request #820 · dashpay/rust-dashcore

xdustinface · 2026-06-24T06:52:21Z

The commit-time batch rescan re-matches scripts derived during block processing against the active batches, but BlockMatchTracker::track subtracted wallets that already had a block applied, so a block processed before those scripts existed was silently skipped. Scripts derived from processing a block could therefore never trigger a re-examination of any earlier block in the same batch, and partially recognized blocks stayed partial.

Add track_for_new_scripts, which keeps the in-flight accounting of track but ignores processed records, and use it in rescan_batch: every match there is by construction against scripts that did not exist at first processing. Re-application is safe because wallet-side block processing is idempotent (transaction dedup, outpoint-keyed UTXO insertion, balance recomputed from the UTXO set). Termination is preserved since the rescan only queries newly collected scripts, and re-processing a block whose outputs are already recognized derives nothing further.

The plain scan_batch path keeps the processed-record skip, late-added wallets still get their residual pass through plain track.

Summary by CodeRabbit

New Features
- Wallet sync now detects payments to newly derived addresses during rescans, helping previously missed transactions appear after script/address discovery expands.
- Historical syncing is more reliable for large batches of fresh addresses, even beyond the usual address gap window.
Bug Fixes
- Fixed an issue where blocks already seen could be skipped during rescan, which could prevent new wallet matches from being counted and delay balance updates.

The commit-time batch rescan re-matches scripts derived during block processing against the active batches, but `BlockMatchTracker::track` subtracted wallets that already had a block applied, so a block processed before those scripts existed was silently skipped. Scripts derived from processing a block could therefore never trigger a re-examination of any earlier block in the same batch, and partially recognized blocks stayed partial. Add `track_for_new_scripts`, which keeps the in-flight accounting of `track` but ignores processed records, and use it in `rescan_batch`: every match there is by construction against scripts that did not exist at first processing. Re-application is safe because wallet-side block processing is idempotent (transaction dedup, outpoint-keyed UTXO insertion, balance recomputed from the UTXO set). Termination is preserved since the rescan only queries newly collected scripts, and re-processing a block whose outputs are already recognized derives nothing further. The plain `scan_batch` path keeps the processed-record skip, late-added wallets still get their residual pass through plain `track`.

One transaction pays a run of consecutive fresh addresses reaching past the gap window, mined before the client ever starts. The historical scan must climb the burst via fixpoint re-scanning and credit every output. Adds `DashCoreNode::send_many` so a regtest wallet can pay many addresses in a single transaction.

coderabbitai · 2026-06-24T06:55:45Z

📝 Walkthrough

Walkthrough

Adds track_for_new_scripts to BlockMatchTracker, which re-queues blocks without an AlreadyProcessed path. Wires this into FiltersManager::rescan_batch for script-driven re-processing. Adds a send_many RPC helper to DashCoreNode and an integration test validating from-scratch SPV sync of a burst payment beyond the external gap window.

Changes

Gap-window rescan via track_for_new_scripts

Layer / File(s)	Summary
`track_for_new_scripts` primitive and unit tests `dash-spv/src/sync/filters/block_match_tracker.rs`	Adds `BlockMatchTracker::track_for_new_scripts`, returning `NewlyTracked` (inserting into `blocks_remaining`) or `InFlight` but never `AlreadyProcessed`. Extends `track_state_machine` test to assert both outcomes.
`rescan_batch` wiring and unit test `dash-spv/src/sync/filters/manager.rs`	Replaces `tracker.track(...)` with `tracker.track_for_new_scripts(...)` in `rescan_batch`, removes the now-impossible `InFlight` arm, and adds `test_rescan_batch_reprocesses_already_processed_block` asserting a previously processed block is re-queued with `pending_blocks` set to 1.
`send_many` helper and burst-payment integration test `dash-spv/src/test_utils/node.rs`, `dash-spv/tests/dashd_sync/tests_transaction.rs`	Adds `DashCoreNode::send_many` using `serde_json::Map` for multi-output RPC calls. Adds `derive_external_addresses` BIP44 helper and `test_burst_payment_beyond_gap_window_synced_from_scratch` asserting from-scratch sync discovers a pre-mined burst payment beyond the external gap window.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

dashpay/rust-dashcore#781: Directly related — that PR refactored the new_scripts/ScriptBuf pipeline through FiltersManager and rescan_batch, which is the same code path that track_for_new_scripts is now wired into.

Suggested labels

ready-for-review

Suggested reviewers

ZocoLini

Poem

🐇 Hop beyond the gap window's wall,
Where old "AlreadyProcessed" could stall,
track_for_new_scripts queues the block anew,
send_many coins to addresses true,
The burst transaction found by SPV's call —
No script derived fresh shall be missed, not at all! 🌿

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: rescanning blocks against newly derived scripts.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/reprocess-blocks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-06-24T06:59:35Z

Codecov Report

❌ Patch coverage is 98.27586% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 73.02%. Comparing base (ceee4a9) to head (368456c).
⚠️ Report is 3 commits behind head on dev.

Files with missing lines	Patch %	Lines
dash-spv/src/sync/filters/manager.rs	96.87%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## dev #820 +/- ##
==========================================
+ Coverage 72.82% 73.02% +0.20% 
==========================================
Files 323 323 Lines 72000 72072 +72 ==========================================
+ Hits 52432 52633 +201 + Misses 19568 19439 -129

Flag	Coverage Δ
core	`76.74% <ø> (ø)`
ffi	`47.34% <ø> (+1.33%)`	⬆️
rpc	`20.00% <ø> (ø)`
spv	`90.24% <98.27%> (+0.06%)`	⬆️
wallet	`71.80% <ø> (ø)`

Files with missing lines	Coverage Δ
dash-spv/src/sync/filters/block_match_tracker.rs	`99.35% <100.00%> (+0.12%)`	⬆️
dash-spv/src/sync/filters/manager.rs	`97.44% <96.87%> (-0.06%)`	⬇️

... and 18 files with indirect coverage changes

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dash-spv/src/test_utils/node.rs`:
- Around line 322-327: The send_many helper is collapsing duplicate destination
addresses when building the serde_json::Map, which can silently change the
intended outputs. Update send_many to detect repeated address strings before
constructing the RPC object, and either reject them with an explicit error or
aggregate their amounts intentionally. Make the behavior clear in send_many and
the amount-to-map building logic so each `(address, amount)` pair is handled
deterministically.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e4d9b656-3aa2-423c-a62f-c0a3fcc1e0ff

📥 Commits

Reviewing files that changed from the base of the PR and between 95a3c8f and 368456c.

📒 Files selected for processing (4)

dash-spv/src/sync/filters/block_match_tracker.rs
dash-spv/src/sync/filters/manager.rs
dash-spv/src/test_utils/node.rs
dash-spv/tests/dashd_sync/tests_transaction.rs

ZocoLini

lgtm so far, I am concern about re-downloading blocks, is it doing that instead of temporally caching them??

ZocoLini · 2026-06-26T09:12:05Z

 blocks_needed.insert(key, wallets);
 }
+ // Never returned by track_for_new_scripts.
 BlockTrackResult::AlreadyProcessed => {}


what about unreachable!() here

xdustinface · 2026-06-26T09:21:18Z

@ZocoLini We don't re-download. Blocks are on disk still.

ZocoLini · 2026-06-26T09:22:33Z

perfect then

…ts (dashpay#846) Scripts derived by gap-limit maintenance mid-sync (CoinJoin index-height inversions being the concrete case) previously had zero backward reach across a batch-commit boundary: `rescan_batch` only reaches `active_batches`, commit prunes the tracker records, and the wallet's advanced `synced_height` filters those heights out of any later match. Outputs paying such scripts in an already-committed range stayed permanently invisible, and a fresh resync hit the same wall. BIP-158 filters are address-independent commitments and are all persisted, so the committed range can be re-tested from storage with no network traffic. `rescan_committed_range` runs at the same commit-time seam as the dashpay#820 forward rescan: it matches only the newly derived scripts against stored filters below the committing batch (chunked, best-effort per chunk), routes hits through the existing `track_for_new_scripts` re-download path, and attributes them to the committing batch so its pending-blocks accounting holds the commit open until the fixpoint drains — scripts derived from re-processed backward blocks feed the next round automatically. Adds `FilterStorage::filter_start_height` to bound the sweep, and adopts the cross-commit repro from `repro/pr3549-rdc` (with header/filter storage seeded to match the production invariant that heights at or below `stored_height` are persisted) as the regression gate; the two dashpay#820 guards stay green. Fixesdashpay#846 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ts (dashpay#846) Scripts derived by gap-limit maintenance mid-sync (CoinJoin index-height inversions being the concrete case) previously had zero backward reach across a batch-commit boundary: `rescan_batch` only reaches `active_batches`, commit prunes the tracker records, and the wallet's advanced `synced_height` filters those heights out of any later match. Outputs paying such scripts in an already-committed range stayed permanently invisible, and a fresh resync hit the same wall. BIP-158 filters are address-independent commitments and are all persisted, so the committed range can be re-tested from storage with no network traffic. `rescan_committed_range` runs at the same commit-time seam as the dashpay#820 forward rescan: it matches only the newly derived scripts against stored filters below the committing batch (chunked, best-effort per chunk), routes hits through the existing `track_for_new_scripts` re-download path, and attributes them to the committing batch so its pending-blocks accounting holds the commit open until the fixpoint drains — scripts derived from re-processed backward blocks feed the next round automatically. Adds `FilterStorage::filter_start_height` to bound the sweep, and adopts the cross-commit repro from `repro/pr3549-rdc` (with header/filter storage seeded to match the production invariant that heights at or below `stored_height` are persisted) as the regression gate; the two Fixesdashpay#846 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ts (#866) * fix(dash-spv): rescan committed filter ranges for newly derived scripts (#846) Scripts derived by gap-limit maintenance mid-sync (CoinJoin index-height inversions being the concrete case) previously had zero backward reach across a batch-commit boundary: `rescan_batch` only reaches `active_batches`, commit prunes the tracker records, and the wallet's advanced `synced_height` filters those heights out of any later match. Outputs paying such scripts in an already-committed range stayed permanently invisible, and a fresh resync hit the same wall. BIP-158 filters are address-independent commitments and are all persisted, so the committed range can be re-tested from storage with no network traffic. `rescan_committed_range` runs at the same commit-time seam as the #820 forward rescan: it matches only the newly derived scripts against stored filters below the committing batch (chunked, best-effort per chunk), routes hits through the existing `track_for_new_scripts` re-download path, and attributes them to the committing batch so its pending-blocks accounting holds the commit open until the fixpoint drains — scripts derived from re-processed backward blocks feed the next round automatically. Adds `FilterStorage::filter_start_height` to bound the sweep, and adopts the cross-commit repro from `repro/pr3549-rdc` (with header/filter storage seeded to match the production invariant that heights at or below `stored_height` are persisted) as the regression gate; the two Fixes#846 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dash-spv): defer the backward sweep to forward quiescence; dedupe match attribution Address review feedback on the committed-range rescan: - The backward sweep over stored history is the expensive direction, and the commit-time fixpoint could re-walk it once per derivation round. Scripts now accumulate on the committing batch (`FiltersBatch::backward_scripts`) while the forward fixpoint converges, and a single combined sweep runs only when the forward direction is quiescent. Blocks it finds re-enter through `collected_scripts`, so only genuinely new scripts get a follow-up sweep — each distinct script crosses the committed range exactly once. (A skip-swept-ranges high-water mark would be unsound here: a range swept for round N's scripts is not covered for round N+1's.) - Extract `queue_new_script_matches` as the shared tail of `rescan_batch` and `rescan_committed_range` (track_for_new_scripts routing, pending_blocks accounting, BlocksNeeded emission) so the two paths cannot silently diverge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(dash-spv): make coinjoin gap repro indices gap-relative after #868 The repros funded indices 40..=51 as "beyond the initial watch window", which was true at the gap of 30 they were written against. #868 widened DEFAULT_COINJOIN_GAP_LIMIT to 100, putting those indices inside the initial window and turning the cross-commit repro trivially green with or without the fix. Express every funded range relative to the gap constant (G+10..=G+21 beyond, 0..=29 within) so the tests keep pinning the beyond-window shapes; verified red on dev (highest_used stalls at Some(29) vs expected Some(121)) and green with the fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…es outputs A transaction processed before its beyond-window output's address was derived records that output as Sent (counterparty) with net_amount equal to the full input value. The gap-limit rescan (#820) re-processes the block and update_utxos heals the account's UTXO set — but confirm_transaction only mutated and re-emitted the record when its context changed, so the record and every persistence mirror built from emitted events kept the born-wrong shape forever. A reload from such a mirror is a visible fund loss (kotlin-sdk TXO-store bug, 2026-08-19). update_utxos now reports every output it recognizes as ours (including outputs skipped for insertion because they are already spent on-chain), and confirm_transaction folds that recognition back into the stored record — role flips (Sent -> Received/Change), net_amount and direction recomputed over the completed details — and returns the corrected record so the caller emits it as an updated-record event. Repro: born_wrong_record_is_corrected_by_gap_rescan drives the real filter -> block -> wallet pipeline with a self-send whose second output pays a beyond-window index, and asserts both the in-memory record and the LAST emitted record carry the corrected ownership. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…rescans after restart Scripts derived during block processing carry a rescan obligation (forward over active batches, backward over the committed range — #820/#846). That obligation lived only in memory: a process death between derivation and the cascade's COMMIT orphaned it, and nothing in a restarted session ever looks below the committed boundary again — outputs paying those scripts stay invisible to the engine forever (the interrupted-restore fund loss, 2026-08-19; on Android, LMK kills make interrupted syncs the common case). The pending-sweep set is now mirrored to metadata storage: persisted the moment scripts enter the manager (before any sweep work), re-seeded into the lowest active batch after a restart (the ordinary commit-time cascade then owns it), and cleared per batch COMMIT against the batch's retired-scripts receipt — the only point that proves the whole fixpoint completed. Opt-in via FiltersManager::with_metadata; managers without it keep the previous in-memory behavior. Repro: interrupted_sweep_is_replayed_after_restart — cross-committed-batch shape, session 1 dropped right after the scripts are derived, session 2 over the same storage recovers the committed-range outputs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…es outputs A transaction processed before its beyond-window output's address was derived records that output as Sent (counterparty) with net_amount equal to the full input value. The gap-limit rescan (#820) re-processes the block and update_utxos heals the account's UTXO set — but confirm_transaction only mutated and re-emitted the record when its context changed, so the record and every persistence mirror built from emitted events kept the born-wrong shape forever. A reload from such a mirror is a visible fund loss (kotlin-sdk TXO-store bug, 2026-08-19). update_utxos now reports every output it recognizes as ours (including outputs skipped for insertion because they are already spent on-chain), and confirm_transaction folds that recognition back into the stored record — role flips (Sent -> Received/Change), net_amount and direction recomputed over the completed details — and returns the corrected record so the caller emits it as an updated-record event. Repro: born_wrong_record_is_corrected_by_gap_rescan drives the real filter -> block -> wallet pipeline with a self-send whose second output pays a beyond-window index, and asserts both the in-memory record and the LAST emitted record carry the corrected ownership. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…rescans after restart Scripts derived during block processing carry a rescan obligation (forward over active batches, backward over the committed range — #820/#846). That obligation lived only in memory: a process death between derivation and the cascade's COMMIT orphaned it, and nothing in a restarted session ever looks below the committed boundary again — outputs paying those scripts stay invisible to the engine forever (the interrupted-restore fund loss, 2026-08-19; on Android, LMK kills make interrupted syncs the common case). The pending-sweep set is now mirrored to metadata storage: persisted the moment scripts enter the manager (before any sweep work), re-seeded into the lowest active batch after a restart (the ordinary commit-time cascade then owns it), and cleared per batch COMMIT against the batch's retired-scripts receipt — the only point that proves the whole fixpoint completed. Opt-in via FiltersManager::with_metadata; managers without it keep the previous in-memory behavior. Repro: interrupted_sweep_is_replayed_after_restart — cross-committed-batch shape, session 1 dropped right after the scripts are derived, session 2 over the same storage recovers the committed-range outputs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

xdustinface added 2 commits June 22, 2026 17:28

coderabbitaiBot requested changes Jun 24, 2026
View reviewed changes

Comment threaddash-spv/src/test_utils/node.rs

coderabbitaiBot approved these changes Jun 24, 2026
View reviewed changes

github-actionsBot added the ready-for-review CodeRabbit has approved this PR label Jun 24, 2026

xdustinface requested a review from ZocoLini June 25, 2026 13:05

ZocoLini approved these changes Jun 26, 2026
View reviewed changes

xdustinface merged commit 1b53458 into devJun 26, 2026
42 checks passed

xdustinface deleted the fix/reprocess-blocks branch June 26, 2026 09:21

Claudius-Maginificent mentioned this pull request Jul 6, 2026
dash-spv: gap-limit scripts derived after a batch commits never re-open the committed range — wallet funds stay invisible (Found-034) #846
Closed

HashEngineering mentioned this pull request Jul 10, 2026
fix(dash-spv): rescan committed filter ranges for newly derived scripts #866
Merged

bfoss765 mentioned this pull request Jul 12, 2026
fix(dash-spv): re-open committed ranges for gap-limit scripts derived after commit (#846) #873
Closed

Navigation Menu

Uh oh!

fix(dash-spv): reprocess blocks on rescan against newly derived scripts - #820

fix(dash-spv): reprocess blocks on rescan against newly derived scripts#820
xdustinface merged 2 commits into
devfrom
fix/reprocess-blocks

xdustinface commented Jun 24, 2026•
edited by coderabbitai Bot
Loading

Uh oh!

coderabbitaiBot commented Jun 24, 2026•
edited
Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Poem

Uh oh!

codecovBot commented Jun 24, 2026

Uh oh!

coderabbitaiBot left a comment

Uh oh!

Uh oh!

ZocoLini left a comment

Uh oh!

ZocoLiniJun 26, 2026

Uh oh!

xdustinface commented Jun 26, 2026

Uh oh!

Uh oh!

ZocoLini commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

xdustinface commented Jun 24, 2026• edited by coderabbitai BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitaiBot commented Jun 24, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Poem

Uh oh!

codecovBot commented Jun 24, 2026

Codecov Report

Uh oh!

coderabbitaiBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZocoLini left a comment

Choose a reason for hiding this comment

Uh oh!

ZocoLiniJun 26, 2026

Choose a reason for hiding this comment

Uh oh!

xdustinface commented Jun 26, 2026

Uh oh!

Uh oh!

ZocoLini commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xdustinface commented Jun 24, 2026•
edited by coderabbitai Bot
Loading

coderabbitaiBot commented Jun 24, 2026•
edited
Loading