Avoid applying onion's channel updates in an observable way by tnull · Pull Request #2666 · lightningdevkit/rust-lightning

tnull · 2023-10-16T11:36:13Z

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak.

Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-16T11:46:09Z

Currently in draft until the approach is clarified. I already started some commits including the ChannelUpdate in PaymentParameters::previously_failed_channels, but don't think this is necessary if we don't reuse any previously failed channels anyways.

codecov-commenter · 2023-10-16T11:48:35Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (1852715) 88.98% compared to head (e64a293) 88.94%.

❗ Current head e64a293 differs from pull request most recent head 1c35255. Consider uploading reports for the commit 1c35255 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2666 +/- ##
==========================================
- Coverage 88.98% 88.94% -0.04% 
==========================================
Files 112 112 Lines 87632 87663 +31 Branches 87632 87663 +31 ==========================================
- Hits 77978 77975 -3 - Misses 7421 7441 +20 - Partials 2233 2247 +14

Files	Coverage Δ
lightning/src/ln/onion_utils.rs	`91.52% <100.00%> (+0.01%)`	⬆️
lightning/src/ln/functional_test_utils.rs	`90.46% <0.00%> (-0.59%)`	⬇️
lightning/src/routing/gossip.rs	`85.89% <61.29%> (-0.40%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-10-16T14:44:46Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

tnull · 2023-10-16T15:44:13Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

Mh, not sure if we want to keep the ability to manually apply updates to a graph around for users knowing what they doing? But if we want to remove it, I think it may make sense to also drop NetworkGraph::handle_network_update and the entire NetworkUpdate type?

TheBlueMatt · 2023-10-16T15:58:12Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....

I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data. I'm a bit torn on applying the "perm fail" case still, because while it exhibits a similar issue, the cost to attack is much higher (you lose out on a lot of fees going forward cause we never route through you now) and its nice to remove graph entries optimistically (we don't currently look at the chain, so the only other reason we remove entries is if they aren't getting any more updates).

tnull · 2023-10-16T16:26:19Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....
I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

Alright, makes sense.

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data.

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

TheBlueMatt · 2023-10-16T16:45:19Z

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Oh, duh, yea.

TheBlueMatt · 2023-10-16T16:46:35Z

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

tnull · 2023-10-16T18:26:15Z

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

Mh, will think about that once more, but currently have no strong opinion on it. I now pushed a commit removing the failure field and dependants. If we're positive we want to go this way, I may look into also removing NetworkUpdate etc.

We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak. Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-19T15:06:54Z

Alright, after more and more backpedaling I now pushed an MVP that just skips application of the NetworkUpdate::ChannelUpdateMessages. I think we should see how to remove them entirely eventually, but this will be a much larger change that requires more coordination beforehand to not lose too much test coverage and not break compatibilty.

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

tnull · 2023-10-16T11:36:13Z

Fixes#2598.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak.

Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-16T11:46:09Z

Currently in draft until the approach is clarified. I already started some commits including the ChannelUpdate in PaymentParameters::previously_failed_channels, but don't think this is necessary if we don't reuse any previously failed channels anyways.

codecov-commenter · 2023-10-16T11:48:35Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (1852715) 88.98% compared to head (e64a293) 88.94%.

❗ Current head e64a293 differs from pull request most recent head 1c35255. Consider uploading reports for the commit 1c35255 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2666 +/- ##
==========================================
- Coverage 88.98% 88.94% -0.04% 
==========================================
Files 112 112 Lines 87632 87663 +31 Branches 87632 87663 +31 ==========================================
- Hits 77978 77975 -3 - Misses 7421 7441 +20 - Partials 2233 2247 +14

Files	Coverage Δ
lightning/src/ln/onion_utils.rs	`91.52% <100.00%> (+0.01%)`	⬆️
lightning/src/ln/functional_test_utils.rs	`90.46% <0.00%> (-0.59%)`	⬇️
lightning/src/routing/gossip.rs	`85.89% <61.29%> (-0.40%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-10-16T14:44:46Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

tnull · 2023-10-16T15:44:13Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

Mh, not sure if we want to keep the ability to manually apply updates to a graph around for users knowing what they doing? But if we want to remove it, I think it may make sense to also drop NetworkGraph::handle_network_update and the entire NetworkUpdate type?

TheBlueMatt · 2023-10-16T15:58:12Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....

I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data. I'm a bit torn on applying the "perm fail" case still, because while it exhibits a similar issue, the cost to attack is much higher (you lose out on a lot of fees going forward cause we never route through you now) and its nice to remove graph entries optimistically (we don't currently look at the chain, so the only other reason we remove entries is if they aren't getting any more updates).

tnull · 2023-10-16T16:26:19Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....
I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

Alright, makes sense.

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data.

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

TheBlueMatt · 2023-10-16T16:45:19Z

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Oh, duh, yea.

TheBlueMatt · 2023-10-16T16:46:35Z

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

tnull · 2023-10-16T18:26:15Z

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

Mh, will think about that once more, but currently have no strong opinion on it. I now pushed a commit removing the failure field and dependants. If we're positive we want to go this way, I may look into also removing NetworkUpdate etc.

We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak. Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-19T15:06:54Z

Alright, after more and more backpedaling I now pushed an MVP that just skips application of the NetworkUpdate::ChannelUpdateMessages. I think we should see how to remove them entirely eventually, but this will be a much larger change that requires more coordination beforehand to not lose too much test coverage and not break compatibilty.

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

tnull · 2023-10-16T11:36:13Z

Fixes#2598.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak.

Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-16T11:46:09Z

Currently in draft until the approach is clarified. I already started some commits including the ChannelUpdate in PaymentParameters::previously_failed_channels, but don't think this is necessary if we don't reuse any previously failed channels anyways.

codecov-commenter · 2023-10-16T11:48:35Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (1852715) 88.98% compared to head (e64a293) 88.94%.

❗ Current head e64a293 differs from pull request most recent head 1c35255. Consider uploading reports for the commit 1c35255 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2666 +/- ##
==========================================
- Coverage 88.98% 88.94% -0.04% 
==========================================
Files 112 112 Lines 87632 87663 +31 Branches 87632 87663 +31 ==========================================
- Hits 77978 77975 -3 - Misses 7421 7441 +20 - Partials 2233 2247 +14

Files	Coverage Δ
lightning/src/ln/onion_utils.rs	`91.52% <100.00%> (+0.01%)`	⬆️
lightning/src/ln/functional_test_utils.rs	`90.46% <0.00%> (-0.59%)`	⬇️
lightning/src/routing/gossip.rs	`85.89% <61.29%> (-0.40%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-10-16T14:44:46Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

tnull · 2023-10-16T15:44:13Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

Mh, not sure if we want to keep the ability to manually apply updates to a graph around for users knowing what they doing? But if we want to remove it, I think it may make sense to also drop NetworkGraph::handle_network_update and the entire NetworkUpdate type?

TheBlueMatt · 2023-10-16T15:58:12Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....

I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data. I'm a bit torn on applying the "perm fail" case still, because while it exhibits a similar issue, the cost to attack is much higher (you lose out on a lot of fees going forward cause we never route through you now) and its nice to remove graph entries optimistically (we don't currently look at the chain, so the only other reason we remove entries is if they aren't getting any more updates).

tnull · 2023-10-16T16:26:19Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....
I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

Alright, makes sense.

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data.

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

TheBlueMatt · 2023-10-16T16:45:19Z

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Oh, duh, yea.

TheBlueMatt · 2023-10-16T16:46:35Z

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

tnull · 2023-10-16T18:26:15Z

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

Mh, will think about that once more, but currently have no strong opinion on it. I now pushed a commit removing the failure field and dependants. If we're positive we want to go this way, I may look into also removing NetworkUpdate etc.

We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak. Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-19T15:06:54Z

Alright, after more and more backpedaling I now pushed an MVP that just skips application of the NetworkUpdate::ChannelUpdateMessages. I think we should see how to remove them entirely eventually, but this will be a much larger change that requires more coordination beforehand to not lose too much test coverage and not break compatibilty.

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

tnull · 2023-10-16T11:36:13Z

Fixes#2598.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak.

Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-16T11:46:09Z

Currently in draft until the approach is clarified. I already started some commits including the ChannelUpdate in PaymentParameters::previously_failed_channels, but don't think this is necessary if we don't reuse any previously failed channels anyways.

codecov-commenter · 2023-10-16T11:48:35Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (1852715) 88.98% compared to head (e64a293) 88.94%.

❗ Current head e64a293 differs from pull request most recent head 1c35255. Consider uploading reports for the commit 1c35255 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2666 +/- ##
==========================================
- Coverage 88.98% 88.94% -0.04% 
==========================================
Files 112 112 Lines 87632 87663 +31 Branches 87632 87663 +31 ==========================================
- Hits 77978 77975 -3 - Misses 7421 7441 +20 - Partials 2233 2247 +14

Files	Coverage Δ
lightning/src/ln/onion_utils.rs	`91.52% <100.00%> (+0.01%)`	⬆️
lightning/src/ln/functional_test_utils.rs	`90.46% <0.00%> (-0.59%)`	⬇️
lightning/src/routing/gossip.rs	`85.89% <61.29%> (-0.40%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-10-16T14:44:46Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

tnull · 2023-10-16T15:44:13Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

Mh, not sure if we want to keep the ability to manually apply updates to a graph around for users knowing what they doing? But if we want to remove it, I think it may make sense to also drop NetworkGraph::handle_network_update and the entire NetworkUpdate type?

TheBlueMatt · 2023-10-16T15:58:12Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....

I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data. I'm a bit torn on applying the "perm fail" case still, because while it exhibits a similar issue, the cost to attack is much higher (you lose out on a lot of fees going forward cause we never route through you now) and its nice to remove graph entries optimistically (we don't currently look at the chain, so the only other reason we remove entries is if they aren't getting any more updates).

tnull · 2023-10-16T16:26:19Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....
I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

Alright, makes sense.

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data.

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

TheBlueMatt · 2023-10-16T16:45:19Z

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Oh, duh, yea.

TheBlueMatt · 2023-10-16T16:46:35Z

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

tnull · 2023-10-16T18:26:15Z

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

Mh, will think about that once more, but currently have no strong opinion on it. I now pushed a commit removing the failure field and dependants. If we're positive we want to go this way, I may look into also removing NetworkUpdate etc.

We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak. Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-19T15:06:54Z

Alright, after more and more backpedaling I now pushed an MVP that just skips application of the NetworkUpdate::ChannelUpdateMessages. I think we should see how to remove them entirely eventually, but this will be a much larger change that requires more coordination beforehand to not lose too much test coverage and not break compatibilty.

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

tnull · 2023-10-16T11:36:13Z

Fixes#2598.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak.

Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-16T11:46:09Z

Currently in draft until the approach is clarified. I already started some commits including the ChannelUpdate in PaymentParameters::previously_failed_channels, but don't think this is necessary if we don't reuse any previously failed channels anyways.

codecov-commenter · 2023-10-16T11:48:35Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (1852715) 88.98% compared to head (e64a293) 88.94%.

❗ Current head e64a293 differs from pull request most recent head 1c35255. Consider uploading reports for the commit 1c35255 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2666 +/- ##
==========================================
- Coverage 88.98% 88.94% -0.04% 
==========================================
Files 112 112 Lines 87632 87663 +31 Branches 87632 87663 +31 ==========================================
- Hits 77978 77975 -3 - Misses 7421 7441 +20 - Partials 2233 2247 +14

Files	Coverage Δ
lightning/src/ln/onion_utils.rs	`91.52% <100.00%> (+0.01%)`	⬆️
lightning/src/ln/functional_test_utils.rs	`90.46% <0.00%> (-0.59%)`	⬇️
lightning/src/routing/gossip.rs	`85.89% <61.29%> (-0.40%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-10-16T14:44:46Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

tnull · 2023-10-16T15:44:13Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

Mh, not sure if we want to keep the ability to manually apply updates to a graph around for users knowing what they doing? But if we want to remove it, I think it may make sense to also drop NetworkGraph::handle_network_update and the entire NetworkUpdate type?

TheBlueMatt · 2023-10-16T15:58:12Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....

I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data. I'm a bit torn on applying the "perm fail" case still, because while it exhibits a similar issue, the cost to attack is much higher (you lose out on a lot of fees going forward cause we never route through you now) and its nice to remove graph entries optimistically (we don't currently look at the chain, so the only other reason we remove entries is if they aren't getting any more updates).

tnull · 2023-10-16T16:26:19Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....
I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

Alright, makes sense.

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data.

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

TheBlueMatt · 2023-10-16T16:45:19Z

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Oh, duh, yea.

TheBlueMatt · 2023-10-16T16:46:35Z

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

tnull · 2023-10-16T18:26:15Z

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

Mh, will think about that once more, but currently have no strong opinion on it. I now pushed a commit removing the failure field and dependants. If we're positive we want to go this way, I may look into also removing NetworkUpdate etc.

We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak. Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-19T15:06:54Z

Alright, after more and more backpedaling I now pushed an MVP that just skips application of the NetworkUpdate::ChannelUpdateMessages. I think we should see how to remove them entirely eventually, but this will be a much larger change that requires more coordination beforehand to not lose too much test coverage and not break compatibilty.

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

tnull · 2023-10-16T11:36:13Z

Fixes#2598.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak.

Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-16T11:46:09Z

Currently in draft until the approach is clarified. I already started some commits including the ChannelUpdate in PaymentParameters::previously_failed_channels, but don't think this is necessary if we don't reuse any previously failed channels anyways.

codecov-commenter · 2023-10-16T11:48:35Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (1852715) 88.98% compared to head (e64a293) 88.94%.

❗ Current head e64a293 differs from pull request most recent head 1c35255. Consider uploading reports for the commit 1c35255 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2666 +/- ##
==========================================
- Coverage 88.98% 88.94% -0.04% 
==========================================
Files 112 112 Lines 87632 87663 +31 Branches 87632 87663 +31 ==========================================
- Hits 77978 77975 -3 - Misses 7421 7441 +20 - Partials 2233 2247 +14

Files	Coverage Δ
lightning/src/ln/onion_utils.rs	`91.52% <100.00%> (+0.01%)`	⬆️
lightning/src/ln/functional_test_utils.rs	`90.46% <0.00%> (-0.59%)`	⬇️
lightning/src/routing/gossip.rs	`85.89% <61.29%> (-0.40%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-10-16T14:44:46Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

tnull · 2023-10-16T15:44:13Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

Mh, not sure if we want to keep the ability to manually apply updates to a graph around for users knowing what they doing? But if we want to remove it, I think it may make sense to also drop NetworkGraph::handle_network_update and the entire NetworkUpdate type?

TheBlueMatt · 2023-10-16T15:58:12Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....

I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data. I'm a bit torn on applying the "perm fail" case still, because while it exhibits a similar issue, the cost to attack is much higher (you lose out on a lot of fees going forward cause we never route through you now) and its nice to remove graph entries optimistically (we don't currently look at the chain, so the only other reason we remove entries is if they aren't getting any more updates).

tnull · 2023-10-16T16:26:19Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....
I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

Alright, makes sense.

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data.

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

TheBlueMatt · 2023-10-16T16:45:19Z

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Oh, duh, yea.

TheBlueMatt · 2023-10-16T16:46:35Z

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

tnull · 2023-10-16T18:26:15Z

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

Mh, will think about that once more, but currently have no strong opinion on it. I now pushed a commit removing the failure field and dependants. If we're positive we want to go this way, I may look into also removing NetworkUpdate etc.

We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak. Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-19T15:06:54Z

Alright, after more and more backpedaling I now pushed an MVP that just skips application of the NetworkUpdate::ChannelUpdateMessages. I think we should see how to remove them entirely eventually, but this will be a much larger change that requires more coordination beforehand to not lose too much test coverage and not break compatibilty.

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

tnull · 2023-10-16T11:36:13Z

Fixes#2598.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak.

Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-16T11:46:09Z

Currently in draft until the approach is clarified. I already started some commits including the ChannelUpdate in PaymentParameters::previously_failed_channels, but don't think this is necessary if we don't reuse any previously failed channels anyways.

codecov-commenter · 2023-10-16T11:48:35Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (1852715) 88.98% compared to head (e64a293) 88.94%.

❗ Current head e64a293 differs from pull request most recent head 1c35255. Consider uploading reports for the commit 1c35255 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2666 +/- ##
==========================================
- Coverage 88.98% 88.94% -0.04% 
==========================================
Files 112 112 Lines 87632 87663 +31 Branches 87632 87663 +31 ==========================================
- Hits 77978 77975 -3 - Misses 7421 7441 +20 - Partials 2233 2247 +14

Files	Coverage Δ
lightning/src/ln/onion_utils.rs	`91.52% <100.00%> (+0.01%)`	⬆️
lightning/src/ln/functional_test_utils.rs	`90.46% <0.00%> (-0.59%)`	⬇️
lightning/src/routing/gossip.rs	`85.89% <61.29%> (-0.40%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-10-16T14:44:46Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

tnull · 2023-10-16T15:44:13Z

Yea, this makes sense to me. Lets just remove the network_update field in PathFailure::OnPath entirely?

Mh, not sure if we want to keep the ability to manually apply updates to a graph around for users knowing what they doing? But if we want to remove it, I think it may make sense to also drop NetworkGraph::handle_network_update and the entire NetworkUpdate type?

TheBlueMatt · 2023-10-16T15:58:12Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....

I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data. I'm a bit torn on applying the "perm fail" case still, because while it exhibits a similar issue, the cost to attack is much higher (you lose out on a lot of fees going forward cause we never route through you now) and its nice to remove graph entries optimistically (we don't currently look at the chain, so the only other reason we remove entries is if they aren't getting any more updates).

tnull · 2023-10-16T16:26:19Z

I mean, I think (a) we think its a terrible idea to apply the update, and would be a lot of work to do so safely (you'd probably have to keep a second copy of the network graph), and (b) we expect to receive the update via the normal gossip network soon anyway, so its not like we're missing out for too long, and (c) we score the channel negatively cause the payment failed (I hope, need to double-check that?) so we shouldn't be retrying over the same channel soon even for a new payment, and (d) probably the network updates will go away in the spec cause its such a bad issue anyway....
I don't really think its worth keeping a bunch of code around for such a rarely-useful case, much better to have less code :)

Alright, makes sense.

I do think we should keep NetworkGraph::handle_network_update just cause who knows where people are getting their gossip data.

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

TheBlueMatt · 2023-10-16T16:45:19Z

Hum, but they'd likely receive gossip data as NodeAnnouncements/ChannelAnnouncements/ChannelUpdates anyways, not NetworkUpdate, which really is only a, to quote its docs, "Update to the NetworkGraph based on payment failure information conveyed via the Onion return packet by a node along the route.", and handle_network_update "Handles any network updates originating from Events."?

Oh, duh, yea.

TheBlueMatt · 2023-10-16T16:46:35Z

Given that it's really just a wrapper type used for the one purpose we're about to drop, it's really tempting to drop all that associated code.

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

tnull · 2023-10-16T18:26:15Z

Yea, I'm just a bit torn on removing from perm failures - it does seem like something worth doing given we don't currently look at the chain to remove after the funding outpoint is spent (and rely on timeouts of the channel_updates). The timeouts are after a week or two, though.

Mh, will think about that once more, but currently have no strong opinion on it. I now pushed a commit removing the failure field and dependants. If we're positive we want to go this way, I may look into also removing NetworkUpdate etc.

We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

If we receive a channel update from an intermediary via a failure onion we shouldn't apply them in a persisted and network-observable way to our network graph, as this might introduce a privacy leak. Here, we therefore avoid applying such updates to our network graph.

tnull · 2023-10-19T15:06:54Z

Alright, after more and more backpedaling I now pushed an MVP that just skips application of the NetworkUpdate::ChannelUpdateMessages. I think we should see how to remove them entirely eventually, but this will be a much larger change that requires more coordination beforehand to not lose too much test coverage and not break compatibilty.

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

tnull marked this pull request as draft October 16, 2023 11:36

tnull force-pushed the 2023-10-observable-update branch from e6ed6af to 5485d1bCompare October 16, 2023 12:24

TheBlueMatt added this to the 0.0.118 milestone Oct 18, 2023

TheBlueMatt self-assigned this Oct 18, 2023

tnull marked this pull request as ready for review October 18, 2023 18:05

TheBlueMatt reviewed Oct 18, 2023
View reviewed changes

Comment threadlightning/src/events/mod.rs

Allow to verify channel updates without applying them
1dffb20
We introduce a new `NetworkGraph::verify_channel_update` method that allows to check whether an update would be applied by `update_channel`.

tnull force-pushed the 2023-10-observable-update branch 2 times, most recently from e64a293 to 8d7aa35Compare October 19, 2023 15:01

tnull force-pushed the 2023-10-observable-update branch from 8d7aa35 to 1c35255Compare October 19, 2023 15:03

TheBlueMatt approved these changes Oct 19, 2023
View reviewed changes

wpaulino approved these changes Oct 19, 2023
View reviewed changes

TheBlueMatt merged commit 6fff3e5 into lightningdevkit:mainOct 19, 2023