public static peel_onion method on OnionMessenger by Evanfeenstra · Pull Request #2599 · lightningdevkit/rust-lightning

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

Evanfeenstra · 2023-09-26T18:36:49Z

Thanks for merging #2583! This is a similar PR, but for the other end (receiving an onion message).

It introduces a new PeeledOnion enum and a new ReceiveError enum.

Of course, open to feedback about the naming, error handling, generics, or anything

codecov-commenter · 2023-09-26T18:46:07Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (989304e) 89.02% compared to head (fe6f166) 88.99%.
Report is 18 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2599 +/- ##
==========================================
- Coverage 89.02% 88.99% -0.04% 
==========================================
Files 112 112 Lines 87168 87437 +269 Branches 87168 87437 +269 ==========================================
+ Hits 77605 77811 +206 - Misses 7327 7387 +60 - Partials 2236 2239 +3

Files	Coverage Δ
lightning/src/onion_message/messenger.rs	`82.04% <80.88%> (+0.64%)`	⬆️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt · 2023-09-26T20:30:08Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Evanfeenstra · 2023-09-26T21:36:01Z

I'm kinda curious why y'all don't want to just create a normal OnionMessager and pass it messages via handle_onion_message? There's nothing really specific about the LDK message handling interfaces that requires them to ride on the standard lightning P2P rails, in fact they're separated traits explicitly to support people not doing that.

Then you would have a make a PeerManager as well, right? And connect a peer, then call read_event, so that peer_connected is called and the OnionMessenger pending_messages is not vacant?

Maybe that's a fine approach... I was just trying to make it as low-level as possible so it can run more efficiently in WASM, embedded, or whatever.

TheBlueMatt · 2023-09-26T21:41:00Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Evanfeenstra · 2023-09-26T22:02:13Z

No, absolutely not. The handle_onion_message function that you edit here is public and can be called directly.

Hmm interesting... what if the caller is not the receiver, but just an intermediate hop? I get Dropping forwarded onion message to disconnected peer when calling handle_onion_message, as the pending_per_peer_msgs is Vacant. So next_onion_message_for_peer() is always None

Not sure the best way to unwrap one layer of an onion message in a free-standing way...

TheBlueMatt · 2023-09-26T22:36:53Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Evanfeenstra · 2023-09-27T17:16:54Z

If you want to forward you'd have to call peer_connected first (though I understood your use-case to not include forwarding? Just doing this at the endpoints). The interface PeerManager uses to talk to the OnionMessenger (and ChannelManager, etc) are all fully public. There's nothing it does that's magic, and anything it normally does can be done manually.

Ok I see what you mean now! Most things can be accomplished by running peer_connected manually.

Our use-case is clients messaging each other via multiple hops ("mixing" through a network of servers for privacy, so server operators cannot track who is chatting). The last routing hop might not always know which recipients it's routing onions to... those clients may connect in the future to fetch stored onions destined for them.

I realize this doesn't make sense in the context of Lightning payments... but currently I don't see a way to unwrap a forwarded onion without knowing the next peer id ahead of time

TheBlueMatt · 2023-09-28T04:02:00Z

Ah, okay, that makes sense. Will pick this back up after 0.0.117 ships, sadly this missed the cut-off.

TheBlueMatt

Basically LGTM, I think, needs rebase tho.

Evanfeenstra · 2023-10-11T19:07:58Z

rebased to main and squashed. Should be good to go now

1 more feedback item outstanding

just noticed something

0.0.118 - Oct 23, 2023 - "Just the Twelve Sinks" API Updates =========== * BOLT12 sending and receiving is now supported as an alpha feature. You may run into unexpected issues and will need to have a direct connection with the offer's blinded path introduction points as messages are not yet routed. We are seeking feedback from early testers (lightningdevkit#2578, lightningdevkit#2039). * `ConfirmationTarget` has been rewritten to provide information about the specific use LDK needs the feerate estimate for, rather than the generic low-, medium-, and high-priority estimates. This allows LDK users to more accurately target their feerate estimates (lightningdevkit#2660). For those wishing to retain their existing behavior, see the table below for conversion. * `ChainHash` is now used in place of `BlockHash` where it represents the genesis block (lightningdevkit#2662). * `lightning-invoice` payment utilities now take a `Deref` to `AChannelManager` (lightningdevkit#2652). * `peel_onion` is provided to statelessly decode an `OnionMessage` (lightningdevkit#2599). * `ToSocketAddrs` + `Display` are now impl'd for `SocketAddress` (lightningdevkit#2636, lightningdevkit#2670) * `Display` is now implemented for `OutPoint` (lightningdevkit#2649). * `Features::from_be_bytes` is now provided (lightningdevkit#2640). For those moving to the new `ConfirmationTarget`, the new variants in terms of the old mempool/low/medium/high priorities are as follows: * `OnChainSweep` = `HighPriority` * `MaxAllowedNonAnchorChannelRemoteFee` = `max(25 * 250, HighPriority * 10)` * `MinAllowedAnchorChannelRemoteFee` = `MempoolMinimum` * `MinAllowedNonAnchorChannelRemoteFee` = `Background - 250` * `AnchorChannelFee` = `Background` * `NonAnchorChannelFee` = `Normal` * `ChannelCloseMinimum` = `Background` Bug Fixes ========= * Calling `ChannelManager::close_channel[_with_feerate_and_script]` on a channel which did not exist would immediately hang holding several key `ChannelManager`-internal locks (lightningdevkit#2657). * Channel information updates received from a failing HTLC are no longer applied to our `NetworkGraph`. This prevents a node which we attempted to route a payment through from being able to learn the sender of the payment. In some rare cases, this may result in marginally reduced payment success rates (lightningdevkit#2666). * Anchor outputs are now properly considered when calculating the amount available to send in HTLCs. This can prevent force-closes in anchor channels when sending payments which overflow the available balance (lightningdevkit#2674). * A peer that sends an `update_fulfill_htlc` message for a forwarded HTLC, then reconnects prior to sending a `commitment_signed` (thus retransmitting their `update_fulfill_htlc`) may result in the channel stalling and being unable to make progress (lightningdevkit#2661). * In exceedingly rare circumstances, messages intended to be sent to a peer prior to reconnection can be sent after reconnection. This could result in undefined channel state and force-closes (lightningdevkit#2663). Backwards Compatibility ======================= * Creating a blinded path to receive a payment then downgrading to LDK prior to 0.0.117 may result in failure to receive the payment (lightningdevkit#2413). * Calling `ChannelManager::pay_for_offer` or `ChannelManager::create_refund_builder` may prevent downgrading to LDK prior to 0.0.118 until the payment times out and has been removed (lightningdevkit#2039). Node Compatibility ================== * LDK now sends a bogus `channel_reestablish` message to peers when they ask to resume an unknown channel. This should cause LND nodes to force-close and broadcast the latest channel state to the chain. In order to trigger this when we wish to force-close a channel, LDK now disconnects immediately after sending a channel-closing `error` message. This should result in cooperative peers also working to confirm the latest commitment transaction when we wish to force-close (lightningdevkit#2658). Security ======== 0.0.118 expands mitigations against transaction cycling attacks to non-anchor channels, though note that no mitigations which exist today are considered robust to prevent the class of attacks. * In order to mitigate against transaction cycling attacks, non-anchor HTLC transactions are now properly re-signed before broadcasting (lightningdevkit#2667). In total, this release features 61 files changed, 3470 insertions, 1503 deletions in 85 commits from 12 authors, in alphabetical order: * Antonio Yang * Elias Rohrer * Evan Feenstra * Fedeparma74 * Gursharan Singh * Jeffrey Czyz * Matt Corallo * Sergi Delgado Segura * Vladimir Fomene * Wilmer Paulino * benthecarman * slanesuke

TheBlueMatt reviewed Oct 10, 2023
View reviewed changes

Comment threadlightning/src/onion_message/messenger.rs Outdated
Comment threadlightning/src/onion_message/messenger.rs Outdated

Evanfeenstra force-pushed the peel-onion branch 3 times, most recently from 5e0990b to 6b168b9Compare October 11, 2023 05:09

valentinewallace self-requested a review October 13, 2023 01:38

valentinewallace reviewed Oct 13, 2023
View reviewed changes

Evanfeenstra force-pushed the peel-onion branch from 72bb977 to be9206cCompare October 14, 2023 20:53

TheBlueMatt mentioned this pull request Oct 15, 2023
BOLT 12 Offers utilities #2578
Merged

valentinewallace previously approved these changes Oct 15, 2023
View reviewed changes

TheBlueMatt reviewed Oct 15, 2023
View reviewed changes

Comment threadlightning/src/onion_message/messenger.rs Outdated

Evanfeenstra force-pushed the peel-onion branch from ffb84e8 to c4e29c0Compare October 15, 2023 20:40

valentinewallace reviewed Oct 16, 2023
View reviewed changes

Comment threadlightning/src/onion_message/messenger.rs Outdated

Evanfeenstra force-pushed the peel-onion branch 2 times, most recently from 0226638 to b7d0008Compare October 16, 2023 22:25

valentinewallace previously approved these changes Oct 16, 2023
View reviewed changes

valentinewallace reviewed Oct 16, 2023
View reviewed changes

Comment threadlightning/src/onion_message/messenger.rs Outdated
Comment threadlightning/src/onion_message/messenger.rs Outdated
Comment threadlightning/src/onion_message/messenger.rs Outdated

Evanfeenstra force-pushed the peel-onion branch from 1ceead9 to de0e102Compare October 17, 2023 00:12

valentinewallace reviewed Oct 17, 2023
View reviewed changes

Comment threadlightning/src/onion_message/messenger.rs Outdated

public static peel_onion method on OnionMessenger
fe6f166

Evanfeenstra force-pushed the peel-onion branch from f01d613 to fe6f166Compare October 17, 2023 21:48

valentinewallace approved these changes Oct 18, 2023
View reviewed changes

TheBlueMatt approved these changes Oct 18, 2023
View reviewed changes

TheBlueMatt merged commit 28602d9 into lightningdevkit:mainOct 18, 2023