Fix double-forward, prefer legacy forward maps by valentinewallace · Pull Request #4289 · lightningdevkit/rust-lightning

valentinewallace · 2025-12-16T22:46:46Z

Addresses a chunk of the feedback from #4227 (review) (tracked in #4280). Splitting it out for ease of review. Partially addresses #4286.

Fix a bug that would cause double-forwarding of inbound HTLCs when using reconstructed forward maps
Prefer legacy forward maps in production while randomly using reconstructed maps in tests for coverage (see commit message)
a few other nits from the aforementioned review

ldk-reviews-bot · 2025-12-16T22:46:49Z

👋 Thanks for assigning @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

valentinewallace · 2025-12-16T22:47:05Z

Going to take another look at this tomorrow before un-drafting it

codecov · 2025-12-17T00:06:18Z

Codecov Report

❌ Patch coverage is 74.47917% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.59%. Comparing base (c9f022b) to head (5a4912c).
⚠️ Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/ln/channelmanager.rs	74.73%	43 Missing and 5 partials ⚠️
lightning/src/ln/channel.rs	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4289 +/- ##
=======================================
Coverage 86.58% 86.59% =======================================
Files 158 158 Lines 102287 102368 +81 Branches 102287 102368 +81 =======================================
+ Hits 88568 88644 +76 - Misses 11304 11311 +7 + Partials 2415 2413 -2

Flag	Coverage Δ
fuzzing	`35.95% <15.42%> (-0.90%)`	⬇️
tests	`85.87% <74.47%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

TheBlueMatt · 2026-01-07T20:06:01Z

 (17, in_flight_monitor_updates, option),
 (19, peer_storage_dir, optional_vec),
 (21, WithoutLength(&self.flow.writeable_async_receive_offer_cache()), required),
+ (23, reconstruct_manager_from_monitors, required),


Oops sorry for the delay. Writing a bool here to determine whether to look at data we're always writing seems quite weird? I'm not sure what the right answer is, but ideally we run tests with both the new and old code. In the past (with block connection) we've taken a somewhat hacky approach of just flipping a coin and using a random value to decide. In general its worked and we haven't seen many cases of flaky tests making their way upstream, but its a bit more annoying for devs. Still, absent a better option (I'm not a huge fan of running the entire test suite twice every time, even running it an extra time in CI kinda sucks...) that seems reasonable.

Agreed, I went with the random option you mention.

We are now still running the entire test suite twice, but not on the same CI run. I think better options are:
Deliberately pick a few tests that sufficient cover the logic, and only run those twice.
Do a more extensive test matrix nightly.

I feel more comfortable with the entire test suite running with the new vs old code rather than selected tests, but adding a whole extra CI run to an already slow CI does suck IMO. So I like this current tradeoff.
How about another environment variable? lol

I think that there will be much more of this for the chan mgr refactor. My draft PRs also assume a safe_channels feature flag and an associated CI job. My preference would be to use the same mechanism here, and accept that while the project is underway, we'll have an additional CI job (a single job for all PRs in the project) that only runs on a single platform. It's not that significant.

valentinewallace · 2026-01-07T20:44:35Z

Addressed feedback, main diff is here. Also pushed some whitespace fixes after.

joostjager · 2026-01-08T08:04:19Z

+ // to ensure the legacy codepaths also have test coverage.
+ #[cfg(not(test))]
+ let reconstruct_manager_from_monitors = false;
+ #[cfg(test)]


What happened to the idea of using the safe_channels flag here, so that we can gate this and all other changes in the chan mgr refactor project, and make it worth doing a separate CI run for it?
Using conditional compilation for the legacy code (not safe_channels) might improve readability. I noticed that I did had to pay some attention to the control flow interventions with for example continue statements when reconstructing. Also makes it easier to delete that code eventually.

valentinewallace · 2026-01-08T17:54:22Z

+ #[cfg(not(test))]
+ let reconstruct_manager_from_monitors = false;
+ #[cfg(test)]
+ let reconstruct_manager_from_monitors = {


@TheBlueMatt pointed out that the way this is currently structured, a future version of LDK that does not write the pending_intercepted_htlcs/forward_htlcs maps will not be able to downgrade to this version of the code, because it only runs the reconstruction logic in tests.
So instead of running reconstruction logic in tests only, we should consider running it if the manager's written version is >= X, where X is a future version where we can assume that the new data is always present and the old data stopped being written.
I'm not sure what version that would be, and I also think we can hold off on this a little until we look into reconstructing more maps. Otherwise we might have some additional complexity (i.e. one var for reconstruct_fwd_maps_from_monitors if version > X, one for reconstruct_claimable_map_from_monitors if version > Y, etc). But worth thinking about and incorporating into upcoming PRs.

How can we add a conditional like that if we don't know yet which version the new data is always present?

valentinewallace · 2026-01-08T19:02:52Z

Discussed offline, going to add an environmental variable to set which manager reconstruction paths to use. Rebased on main to get the changes from #4296

valentinewallace · 2026-01-08T19:32:47Z

Added the environment variable: diff

joostjager

As mentioned before, I reluctantly accept the test input randomization and the reload boolean.

The only thing I'd like to be sure of that I really understand is the upgrade/downgrade plan that was landed on. Ideally the new data would be written and used in the next release. Postponing that another release is a big decision and it seems there is not that much gained from it except for removal of some code that already exists in main.

joostjager · 2026-01-09T12:39:50Z

+ // persist that state, relying on it being up-to-date on restart. Newer versions are moving
+ // towards reducing this reliance on regular persistence of the `ChannelManager`, and instead
+ // reconstruct HTLC/payment state based on `Channel{Monitor}` data if
+ // `reconstruct_manager_from_monitors` is set below. Currently it is only set in tests, randomly


Bringing everything together, would the upgrade/downgrade situation look like this?
Version Read Write Upgrade to version (max) Downgrade to version (min)
0.2 legacy legacy 0.5 -
0.3 legacy legacy+new 0.6 0.2
0.4 legacy+new legacy+new 0.6 0.2
0.5 legacy+new new 0.6 0.4
0.6 new new - 0.4
Also wondering - in relation to the discussion in yesterday's sync meet - how the reconstruct_manager_from_monitors makes for a faster path?

would the upgrade/downgrade situation look like this?
My understanding is that our goal is that 0.3 would support reading new as well, so that 0.5 can downgrade to 0.3 rather than 0.4.
Also wondering - in relation to the discussion in yesterday's sync meet - how the reconstruct_manager_from_monitors makes for a faster path?
So that we get the above. I don't see a reason to want to only allow 0.5 to downgrade to 0.4 rather than 0.3. The code currently doesn't do that but presumably in a followup we could do that?

@valentinewallace and I discussed this more. It seems that having a dedicated flag for signaling that the old maps are not written is better than guessing at a future version number where this is the case.
This currently assumes we'll skip one version before merging the final changes (new read and new write only), but the flexibility remains to wait more versions and enlarge the downgrade window.
Version Read Write Upgrade to version (max) Downgrade to version (min) Set flag
0.2 (current) legacy legacy 0.4 any
0.3 legacy OR new with flag legacy+new any any
0.4 legacy OR new with flag legacy+new any any
0.5 new new any 0.3 X

It seems that having a dedicated flag for signaling that the old maps are not written is better than guessing at a future version number where this is the case.
You mean instead of using SERIALIZATION_VERSION/MIN_SERIALIZATION_VERSION constants you want to use a TLV? I guess that's fine, but it seems much simpler to use the version numbers so that we can also drop some of the legacy crap that is written as non-TLVs that we'll never be writing anymore. Don't really see a reason to avoid that.
The upgrade path looks right to me, though.

I still think it is not a great idea to assume things about a specific future version number.

Would that look like this?
Version Read Write Upgrade to version (max) Downgrade to version (min) SER_VER MIN_SER_VER
0.2 (current) legacy legacy 0.4 any 1 1
0.3 legacy OR new with flag legacy+new any any 2 1
0.4 legacy OR new with flag legacy+new any any 2 1
0.5 new new any 0.3 3 2
We are wondering what happens if we need to bump in between for a different reason.
There is also the issue that the new format is really a per-htlc format change, not overall chan mgr version.

joostjager · 2026-01-09T12:41:06Z

+ #[cfg(not(test))]
+ let reconstruct_manager_from_monitors = false;
+ #[cfg(test)]
+ let reconstruct_manager_from_monitors = {


How can we add a conditional like that if we don't know yet which version the new data is always present?

joostjager · 2026-01-09T13:41:34Z

You may want to update the PR title and description with details

ldk-reviews-bot · 2026-01-10T00:01:03Z

🔔 1st Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-01-12T00:01:41Z

🔔 2nd Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Necessary for the next commit and makes it easier to read.

We recently began reconstructing ChannelManager::decode_update_add_htlcs on startup, using data present in the Channels. However, we failed to prune HTLCs from this rebuilt map if a given HTLC was already forwarded to the outbound edge (we pruned correctly if the outbound edge was a closed channel, but not otherwise). Here we fix this bug that would have caused us to double-forward inbound HTLC forwards.

No need to iterate through all entries in the map, we can instead pull out the specific entry that we want.

We are working on removing the requirement of regularly persisting the ChannelManager, and as a result began reconstructing the manager's forwards maps from Channel data on startup in a recent PR, see cb398f6 and parent commits. At the time, we implemented ChannelManager::read to prefer to use the newly reconstructed maps, partly to ensure we have test coverage of the new maps' usage. This resulted in a lot of code that would deduplicate HTLCs that were present in the old maps to avoid redundant HTLC handling/duplicate forwards, adding extra complexity. Instead, always use the old maps in prod, but randomly use the newly reconstructed maps in testing, to exercise the new codepaths (see reconstruct_manager_from_monitors in ChannelManager::read).

joostjager

Assuming the upgrade story will be refined in the follow up, all my comments have been addressed.

This was referenced Dec 16, 2025
Reconstruct ChannelManager forwarded HTLCs maps from Channels #4227
Merged
Stale ChannelManager causes channel force-closures #4286
Open

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from 89f5d07 to c6bb096Compare December 17, 2025 20:25

valentinewallace commented Dec 17, 2025
View reviewed changes

Comment threadlightning/src/ln/channelmanager.rs Outdated

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from c6bb096 to 425747eCompare December 17, 2025 21:13

joostjager reviewed Dec 18, 2025
View reviewed changes

Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning/src/ln/channelmanager.rs Outdated

valentinewallace mentioned this pull request Jan 6, 2026
Prevent HTLC double-forwards, prune forwarded onions #4303
Merged
2 tasks

valentinewallace added this to the 0.3 milestone Jan 6, 2026

TheBlueMatt reviewed Jan 7, 2026
View reviewed changes

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch 3 times, most recently from 7b858c1 to 71062f2Compare January 7, 2026 20:42

valentinewallace marked this pull request as ready for review January 7, 2026 20:44

ldk-reviews-bot requested a review from wpaulino January 7, 2026 20:45

valentinewallace requested review from TheBlueMatt and joostjager and removed request for wpaulino January 7, 2026 20:49

joostjager reviewed Jan 8, 2026
View reviewed changes

valentinewallace commented Jan 8, 2026
View reviewed changes

valentinewallace self-assigned this Jan 8, 2026

valentinewallace added this to Weekly Goals Jan 8, 2026

valentinewallace moved this to Goal: Merge in Weekly Goals Jan 8, 2026

valentinewallace added the weekly goal Someone wants to land this this week label Jan 8, 2026

Remove unnecessary update_add clone on Channel ser
7c97f3c

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from 71062f2 to f1ea7bbCompare January 8, 2026 19:01

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from f1ea7bb to f245139Compare January 8, 2026 19:31

valentinewallace requested a review from joostjager January 8, 2026 19:32

joostjager reviewed Jan 9, 2026
View reviewed changes

valentinewallace changed the title ~~Follow-ups to #4227 (Part 1)~~Fix double-forward, prefer legacy forward mapsJan 12, 2026

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from f245139 to 17b562cCompare January 12, 2026 21:02

valentinewallace added 5 commits January 12, 2026 16:13

Move is_chan_closed check into loop
2a1273b
Necessary for the next commit and makes it easier to read.

Optimize dedup_decode_update_add_htlcs
03882bd
No need to iterate through all entries in the map, we can instead pull out the specific entry that we want.

Update outdated comment due to var renames
5a4912c

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from 17b562c to 5a4912cCompare January 12, 2026 21:15

valentinewallace requested a review from joostjager January 13, 2026 14:51

joostjager approved these changes Jan 13, 2026
View reviewed changes

TheBlueMatt approved these changes Jan 13, 2026
View reviewed changes

TheBlueMatt merged commit c5d7b13 into lightningdevkit:mainJan 13, 2026
19 of 20 checks passed

github-project-automationBot moved this from Goal: Merge to Done in Weekly Goals Jan 13, 2026

TheBlueMatt mentioned this pull request Jan 14, 2026
#4227 Followups #4280
Closed

Version	Read	Write	Upgrade to version (max)	Downgrade to version (min)
0.2	legacy	legacy	0.5	-
0.3	legacy	legacy+new	0.6	0.2
0.4	legacy+new	legacy+new	0.6	0.2
0.5	legacy+new	new	0.6	0.4
0.6	new	new	-	0.4

Version	Read	Write	Upgrade to version (max)	Downgrade to version (min)	Set flag
0.2 (current)	legacy	legacy	0.4	any
0.3	legacy OR new with flag	legacy+new	any	any
0.4	legacy OR new with flag	legacy+new	any	any
0.5	new	new	any	0.3	X

Uh oh!

Conversation

valentinewallace commented Dec 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldk-reviews-bot commented Dec 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valentinewallace commented Dec 16, 2025

Uh oh!

codecovBot commented Dec 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valentinewallaceJan 8, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joostjagerJan 8, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valentinewallace commented Jan 7, 2026

Uh oh!

joostjagerJan 8, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valentinewallace commented Jan 8, 2026

Uh oh!

valentinewallace commented Jan 8, 2026

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheBlueMattJan 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joostjagerJan 12, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joostjagerJan 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joostjager commented Jan 9, 2026

Uh oh!

ldk-reviews-bot commented Jan 10, 2026

Uh oh!

ldk-reviews-bot commented Jan 12, 2026

Uh oh!

joostjager left a comment

valentinewallace commented Dec 16, 2025•
edited
Loading

ldk-reviews-bot commented Dec 16, 2025•
edited
Loading

codecovBot commented Dec 17, 2025•
edited
Loading

valentinewallaceJan 8, 2026•
edited
Loading

joostjagerJan 8, 2026•
edited
Loading

joostjagerJan 8, 2026•
edited
Loading

TheBlueMattJan 11, 2026•
edited
Loading

joostjagerJan 12, 2026•
edited
Loading

joostjagerJan 14, 2026•
edited
Loading