Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Add a new `RecipientOnionFields` and replace `PaymentSecret` with it by TheBlueMatt · Pull Request #2139 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add a new `RecipientOnionFields` and replace `PaymentSecret` with it by TheBlueMatt · Pull Request #2139 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add a new `RecipientOnionFields` and replace `PaymentSecret` with it by TheBlueMatt · Pull Request #2139 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Add a new `RecipientOnionFields` and replace `PaymentSecret` with it by TheBlueMatt · Pull Request #2139 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add a new `RecipientOnionFields` and replace `PaymentSecret` with it by TheBlueMatt · Pull Request #2139 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add a new `RecipientOnionFields` and replace `PaymentSecret` with it by TheBlueMatt · Pull Request #2139 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Add a new `RecipientOnionFields` and replace `PaymentSecret` with it by TheBlueMatt · Pull Request #2139 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add a new RecipientOnionFields and replace PaymentSecret with it - #2139

Merged
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors
Apr 7, 2023
Merged

Add a new RecipientOnionFields and replace PaymentSecret with it#2139
TheBlueMatt merged 5 commits into
lightningdevkit:mainfrom
TheBlueMatt:2023-03-metadata-prefactors

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This is the bulk of #2127 by LoC changes, but none of the actual work - it imply adds the RecipientOnionFields struct and pipes it through all the send calls. #2127 will then be relatively simple in just adding the metadata feature/field and putting it in the onion.

@tnull
tnull self-requested a review March 30, 2023 08:19
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from 855e240 to 3323137CompareMarch 30, 2023 18:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after CI is fixed and tnull takes a look

Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated
Comment on lines +240 to +241
/// If you do not have one, the [`Route`] you pay over must not contain multiple paths as
/// multi-path payments require a recipient-provided secret.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, do we still plan to do #1222? I may have recently suggested it to a new contributor

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not, but someone totally should!

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, broke bench like always lol. The other issues are upstream cause futures broke us.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 3 times, most recently from 9ee94f2 to c86099cCompareMarch 30, 2023 22:31
@codecov-commenter

codecov-commenter commented Mar 31, 2023

Copy link
Copy Markdown

Codecov Report

Patch coverage: 98.46% and project coverage change: +0.25 🎉

Comparison is base (3b8bf93) 91.35% compared to head (c86099c) 91.60%.

❗ Current head c86099c differs from pull request most recent head f368f80. Consider uploading reports for the commit f368f80 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@ Coverage Diff @@## main #2139 +/- ##
==========================================
+ Coverage 91.35% 91.60% +0.25% 
==========================================
Files 102 102 Lines 49909 52715 +2806 Branches 49909 52715 +2806 ==========================================
+ Hits 45592 48289 +2697 - Misses 4317 4426 +109 
Impacted FilesCoverage Δ
lightning/src/ln/channel.rs90.04% <ø> (-0.04%)⬇️
lightning/src/ln/functional_test_utils.rs94.35% <33.33%> (+1.76%)⬆️
lightning-invoice/src/payment.rs83.01% <66.66%> (+0.16%)⬆️
lightning/src/ln/channelmanager.rs89.83% <96.77%> (+0.70%)⬆️
lightning/src/ln/outbound_payment.rs90.68% <98.33%> (+0.15%)⬆️
lightning-invoice/src/utils.rs97.01% <100.00%> (-0.05%)⬇️
lightning/src/chain/chainmonitor.rs94.65% <100.00%> (-0.08%)⬇️
lightning/src/chain/channelmonitor.rs94.46% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs98.61% <100.00%> (+0.01%)⬆️
lightning/src/ln/functional_tests.rs98.92% <100.00%> (+0.69%)⬆️
... and 7 more

... and 11 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@TheBlueMattTheBlueMatt added this to the 0.0.115 milestone Apr 3, 2023
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from c86099c to c77a59dCompareApril 3, 2023 22:24
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Rebased.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, only a few nits after a first pass.

CI is still sad though.

Comment threadlightning-invoice/src/payment.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

/// Information which is provided, encrypted, to the payment recipient when sending HTLCs.
///
/// This should generally be constructed with data communicated to us from the recipient (via some

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is very vague and I'm actually not sure what it tries to tell me. How is which data communicated to us? What forms of invoices are there and which should I expect..?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BOLT11 or BOLT12?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. Maybe the sentence can even be dropped, but in its current form I don't find it to add any value: "something might happen sometimes in a variety of ways, possibly".

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it stronger, I mean the point is that you're definitely getting the info from an invoice unless you're doing something super custom (or some "extension" protocol).

Comment threadlightning/src/ln/channelmanager.rs Outdated
/// form of invoice).
#[derive(Clone)]
pub struct RecipientOnionFields {
/// The [`PaymentSecret`] which is an arbitrary 32 bytes provided by the recipient for us to

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "(..) which consists of 32 random bytes that were provided by the recipient and should be included in the onion."

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing that says it has to be random? In fact its not in LDK.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, still the sentence could use some restructuring and "an 32 bytes provided" is simply missing a word, no?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, needed to drop which.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fb26548 to f129e9aCompareApril 4, 2023 17:26
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Oops, realized there were some small fixups from the parent PR that didnt end up here, pushed those. CI should fail on the each-commit-builds test but should pass after squash.

@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch 2 times, most recently from fa3c946 to f368f80CompareApril 4, 2023 17:42
@tnull

tnull commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

LGTM, will give it a final round after squash.

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after squash

Comment threadlightning/src/ln/channelmanager.rs
Many of the fields in `HTLCSource::OutboundRoute` are used to
rebuild the pending-outbound-payment map on reload if the
`ChannelManager` was not serialized though `ChannelMonitor`(s)
were after an HTLC was sent. As of 0.0.114, however, such payments
are not retryable without allowing them to fail and doing a full,
fresh, send.
Thus, some of the fields can be safely removed - we only really
care about having enough information to provide the user a failure
event, not being able to retry.
Here we drop one such field - the `payment_secret`, making our
`ChannelMonitorUpdate`s another handful of bytes smaller.
This moves the public payment sending API from passing an explicit
`PaymentSecret` to a new `RecipientOnionFields` struct (which
currently only contains the `PaymentSecret`). This gives us
substantial additional flexibility as we look at add both
`PaymentMetadata`, a new (well, year-or-two-old) BOLT11 invoice
extension to provide additional data sent to the recipient.
In the future, we should also add the ability to add custom TLV
entries in the `RecipientOnionFields` struct.
While most lightning nodes don't (currently) support providing a
payment secret or payment metadata for spontaneous payments,
there's no specific technical reason why we shouldn't support
sending those fields to a recipient.
Further, when we eventually move to allowing custom TLV entries in
the recipient's onion TLV stream, we'll want to support it for
spontaneous payments as well.
Here we simply add the new `RecipientOnionFields` struct as an
argument to the spontaneous payment send methods. We don't yet
plumb it through the payment sending logic, which will come when we
plumb the new struct through the sending logic to replace the
existing payment secret arguments.
This passes the new `RecipientOnionFields` through the internal
sending APIs, ensuring we have access to the full struct when we
go to construct the sending onion so that we can include any new
fields added there.
@TheBlueMatt
TheBlueMattforce-pushed the 2023-03-metadata-prefactors branch from f368f80 to 36235c3CompareApril 5, 2023 16:28
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and added one commit to correct events docs.

@TheBlueMatt
TheBlueMatt merged commit 1016e1f into lightningdevkit:mainApr 7, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes"
API Updates
===========
* The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107).
* Attempting to claim an un-expired payment on a channel which has closed no
longer fails. The expiry time of payments is exposed via
`PaymentClaimable::claim_deadline` (lightningdevkit#2148).
* `payment_metadata` is now supported in `Invoice` deserialization, sending,
and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127).
* `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142).
* BOLT12 messages now support stateless generation and validation (lightningdevkit#1989).
* The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161).
* Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138).
* `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085).
* A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098).
* `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136).
* `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079).
* `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094).
* `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023).
* The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206).
Backwards Compatibility
=======================
* Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no
longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139).
* `Event::PaymentPathFailed::retry` was removed and will always be `None` for
payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063).
* `Route`s and `PaymentParameters` with blinded path information will not be
readable on prior versions of LDK. Such objects are not currently constructed
by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146).
* Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a
`ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is
in general unsupported, and included here only for completeness.
Bug Fixes
=========
* Fixed a case where `process_events_async` may `poll` a `Future` which has
already completed (lightningdevkit#2081).
* Fixed deserialization of `u16` arrays. This bug may have previously corrupted
the historical buckets in a `ProbabilisticScorer`. Users relying on the
historical buckets may wish to wipe their scorer on upgrade to remove corrupt
data rather than waiting on it to decay (lightningdevkit#2191).
* The `process_events_async` task is now `Send` and can thus be polled on a
multi-threaded runtime (lightningdevkit#2199).
* Fixed a missing macro export causing
`impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091).
* Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187)
* Fix an issue where the `background-processor` would not wake when a
`ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090).
* Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145).
* `Router` calls from the `ChannelManager` now call `find_route_with_id` rather
than `find_route`, as was intended and described in the API (lightningdevkit#2092).
* Ensure `process_events_async` always exits if any sleep future returns true,
not just if all sleep futures repeatedly return true (lightningdevkit#2145).
* `channel_update` messages no longer set the disable bit unless the peer has
been disconnected for some time. This should resolve cases where channels are
disabled for extended periods of time (lightningdevkit#2198).
* We no longer remove CLN nodes from the network graph for violating the BOLT
spec in some cases after failing to pay through them (lightningdevkit#2220).
* Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172).
* `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114)
* Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158).
Security
========
* Transaction re-broadcasting is now substantially more aggressive, including a
new regular rebroadcast feature called on a timer from the
`background-processor` or from `ChainMonitor::rebroadcast_pending_claims`.
This should substantially increase transaction confirmation reliability
without relying on downstream `TransactionBroadcaster` implementations for
rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208).
* Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a
privacy vulnerability which allows an intermediate node on the path to
discover the final destination for a payment (lightningdevkit#2062).
In total, this release features 110 files changed, 11928 insertions, 6368
deletions in 215 commits from 21 authors, in alphabetical order:
* Advait
* Alan Cohen
* Alec Chen
* Allan Douglas R. de Oliveira
* Arik Sosman
* Elias Rohrer
* Evan Feenstra
* Jeffrey Czyz
* John Cantrell
* Lucas Soriano del Pino
* Marc Tyndel
* Matt Corallo
* Paul Miller
* Steven
* Steven Williamson
* Steven Zhao
* Tony Giorgio
* Valentine Wallace
* Wilmer Paulino
* benthecarman
* munjesi
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@codecov-commenter@tnull@valentinewallace