Fix race condition causing async payment failure by valentinewallace · Pull Request #4106 · lightningdevkit/rust-lightning

valentinewallace · 2025-09-22T18:02:07Z

As the LSP of an async sender, when we receive an update_add with the hold_htlc flag set, after its onion is decoded we transition the pending HTLC to the ChannelManager::pending_intercepted_htlcs. However, if we receive the release_held_htlc message from the receiver before we've had a chance to make this transition, we'll fail to release the HTLC and it will sit in the pending intercepts map until it is failed backwards.

To fix this race condition, if we receive release_held_htlc from the recipient we'll not only check the pending_intercepted_htlcs map for the presence of this HTLC but also check the map where we keep HTLCs prior to their onions being decoded.

ldk-reviews-bot · 2025-09-22T18:02:09Z

👋 Thanks for assigning @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

codecov · 2025-09-22T18:12:44Z

Codecov Report

❌ Patch coverage is 92.85714% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.53%. Comparing base (e82ef2c) to head (ade1f34).
⚠️ Report is 92 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/ln/channelmanager.rs	89.23%	7 Missing ⚠️
lightning/src/ln/async_payments_tests.rs	97.56%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4106 +/- ##
=========================================
Coverage 88.53% 88.53% =========================================
Files 175 179 +4 Lines 132702 134407 +1705 Branches 132702 134407 +1705 =========================================
+ Hits 117484 118994 +1510 - Misses 12618 12658 +40 - Partials 2600 2755 +155

Flag	Coverage Δ
fuzzing	`21.78% <26.76%> (?)`
tests	`88.37% <92.85%> (-0.16%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ldk-reviews-bot · 2025-09-23T07:52:48Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

joostjager · 2025-09-24T15:08:20Z

+ lock_in_htlc_for_static_invoice(&static_invoice_om, peer_id, sender, sender_lsp);
+
+	// The LSP has not transitioned the HTLC to the intercepts map internally because
+	// process_pending_htlc_forwards has not been called.


Manual message passing is really an advantage here.

ldk-reviews-bot · 2025-09-24T16:01:14Z

✅ Added second reviewer: @jkczyz

joostjager · 2025-09-24T16:10:55Z

This PR also unlocks more e2e testing in ldk-node.

valentinewallace · 2025-09-24T16:29:02Z

@jkczyz understandable if it doesn't make sense for you to review here, feel free to reroll the bot's assignment

TheBlueMatt · 2025-09-24T19:56:02Z

 macro_rules! handle_monitor_update_completion {
 	($self: ident, $peer_state_lock: expr, $peer_state: expr, $per_peer_state_lock: expr, $chan: expr) => { {
 let channel_id = $chan.context.channel_id();
+ let short_channel_id = $chan.funding.get_short_channel_id().unwrap_or($chan.context().outbound_scid_alias());


Let's just change decode_update_add_htlcs to use the outbound alias? That won't change and also it looks like the SCID itself is basically unused, so there's no reason to use the real one.

Oh fuck this is a mess wrt splicing. Please definitely do this #4121

Should be addressed, though we don't move to (PublicKey, ChannelId) here

Since we have added/are adding splicing support, the scid of a channel is liable to change post-splice. Some maps in the ChannelManager are keyed by the scid of a channel, which is an issue now -- if we forward an HTLC from a channel and then splice that channel before the HTLC is resolved, we'll end up with an HTLC source with an scid that doesn't correspond to any open channel. This may result in loss of the HTLC resolution. The outbound scid alias of a channel is stable even post-splice, so for the short term here we switch to using that instead. In the medium term we should update these maps to use (PublicKey, ChannelId) like everything else. We don't always use the alias for outbound forwarded HTLCs, since we tend to use whatever outbound scid is in the onion. That's fine because we properly handle the case where the outbound channel cannot be found; the main problem is in inbound HTLCs and forgetting their resolution.

As the LSP of an async sender, when we receive an update_add with the hold_htlc flag set, after its onion is decoded we transition the pending HTLC to the ChannelManager::pending_intercepted_htlcs. However, if we receive the release_held_htlc message from the receiver *before* we've had a chance to make this transition, we'll fail to release the HTLC and it will sit in the pending intercepts map until it is failed backwards. To fix this race condition, if we receive release_held_htlc from the recipient we'll not only check the pending_intercepted_htlcs map for the presence of this HTLC but also check the map where we keep HTLCs prior to their onions being decoded.

valentinewallace · 2025-09-24T21:58:22Z

 let mut decode_update_add_htlcs = None;
 if !pending_update_adds.is_empty() {
- decode_update_add_htlcs = Some((short_channel_id, pending_update_adds));
+ decode_update_add_htlcs = Some((outbound_scid_alias, pending_update_adds));


I believe this is the only place we key to-be-decoded update_adds with an scid, which should happen for all inbound HTLCs. So I think the source scid will always be the alias now.

joostjager

Second commit re-acked. First commit I need some context, which I'll get offline. Will approve optimistically for now, to keep moving.

Handle `ReleaseHeldHtlc` messages that arrive before the sender-side LSP has even queued the held HTLC for onion decoding. Unlike lightningdevkit#4106, which covers releases arriving after the HTLC is in `decode_update_add_htlcs` but before it reaches `pending_intercepted_htlcs`, this preserves releases that arrive one step earlier and would otherwise be dropped as HTLC not found. Co-Authored-By: HAL 9000 Signed-off-by: Elias Rohrer <dev@tnull.de>

Handle `ReleaseHeldHtlc` messages that arrive before the sender-side LSP has even queued the held HTLC for onion decoding. Unlike lightningdevkit#4106, which covers releases arriving after the HTLC is in `decode_update_add_htlcs` but before it reaches `pending_intercepted_htlcs`, this preserves releases that arrive one step earlier and would otherwise be dropped as HTLC not found. Co-Authored-By: HAL 9000 Co-Authored-By: Elias Rohrer <dev@tnull.de>

valentinewallace requested a review from joostjager September 22, 2025 18:02

graphite-appBot reviewed Sep 22, 2025
View reviewed changes

Comment threadlightning/src/ln/channelmanager.rs Outdated

valentinewallace force-pushed the 2025-08-async-sender-fix-race branch from 1be75bd to 5ed1b9fCompare September 22, 2025 20:03

valentinewallace added the weekly goal Someone wants to land this this week label Sep 22, 2025

TheBlueMatt added this to the 0.2 milestone Sep 22, 2025

joostjager reviewed Sep 23, 2025
View reviewed changes

Comment threadlightning/src/blinded_path/message.rs Outdated

joostjager previously approved these changes Sep 24, 2025
View reviewed changes

valentinewallace dismissed joostjager’s stale review via a041894September 24, 2025 14:27

valentinewallace force-pushed the 2025-08-async-sender-fix-race branch from 5ed1b9f to a041894Compare September 24, 2025 14:27

graphite-appBot reviewed Sep 24, 2025
View reviewed changes

Comment threadlightning/src/ln/channelmanager.rs

joostjager previously approved these changes Sep 24, 2025
View reviewed changes

ldk-reviews-bot requested a review from jkczyz September 24, 2025 16:01

elnosh approved these changes Sep 24, 2025
View reviewed changes

TheBlueMatt requested review from TheBlueMatt and removed request for jkczyz September 24, 2025 19:50

TheBlueMatt reviewed Sep 24, 2025
View reviewed changes

valentinewallace added 2 commits September 24, 2025 17:30

valentinewallace dismissed joostjager’s stale review via ade1f34September 24, 2025 21:51

valentinewallace force-pushed the 2025-08-async-sender-fix-race branch from a041894 to ade1f34Compare September 24, 2025 21:51

valentinewallace commented Sep 24, 2025
View reviewed changes

valentinewallace requested a review from TheBlueMatt September 24, 2025 22:11

TheBlueMatt approved these changes Sep 25, 2025
View reviewed changes

TheBlueMatt requested a review from joostjager September 25, 2025 01:16

joostjager approved these changes Sep 25, 2025
View reviewed changes

joostjager merged commit 9514637 into lightningdevkit:mainSep 25, 2025
25 checks passed

tnull mentioned this pull request Apr 6, 2026
Fix async release before HTLC decode #4542
Closed

valentinewallace mentioned this pull request Apr 8, 2026
Fix async release before HTLC decode #4548
Merged

Uh oh!

Conversation

valentinewallace commented Sep 22, 2025

Uh oh!

ldk-reviews-bot commented Sep 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecovBot commented Sep 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

ldk-reviews-bot commented Sep 23, 2025

Uh oh!

Uh oh!

Uh oh!

joostjagerSep 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ldk-reviews-bot commented Sep 24, 2025

Uh oh!

joostjager commented Sep 24, 2025

Uh oh!

valentinewallace commented Sep 24, 2025

Uh oh!

TheBlueMattSep 24, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMattSep 24, 2025

Choose a reason for hiding this comment

Uh oh!

valentinewallaceSep 24, 2025

Choose a reason for hiding this comment

Uh oh!

valentinewallaceSep 24, 2025

Choose a reason for hiding this comment

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ldk-reviews-bot commented Sep 22, 2025•
edited
Loading

codecovBot commented Sep 22, 2025•
edited
Loading