Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Remove exclusive reference and generic from CommitmentTransaction API - #3689

Merged
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared
Apr 16, 2025
Merged

Remove exclusive reference and generic from CommitmentTransaction API#3689
TheBlueMatt merged 1 commit into
lightningdevkit:mainfrom
tankyleo:small-n-squared

Conversation

@tankyleo

@tankyleotankyleo commented Mar 29, 2025

Copy link
Copy Markdown
Contributor
 Remove exclusive reference and generic from `CommitmentTransaction` API
This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.

@ldk-reviews-bot

ldk-reviews-bot commented Mar 29, 2025

Copy link
Copy Markdown

👋 Thanks for assigning @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@tankyleotankyleo changed the title Remove generic and mutable references from commitment transaction APIRemove generic and mutable references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and mutable references from CommitmentTransaction APIRemove generic and exclusive references from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo changed the title Remove generic and exclusive references from CommitmentTransaction APIRemove generic and exclusive reference from CommitmentTransaction APIMar 29, 2025
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from c8a753e to b04cc76CompareMarch 29, 2025 01:07
@tankyleotankyleo changed the title Remove generic and exclusive reference from CommitmentTransaction APIRemove exclusive reference and generic from CommitmentTransaction APIMar 29, 2025
@tankyleotankyleo mentioned this pull request Mar 10, 2025
24 tasks
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from f4a3cc3 to acad6d8CompareApril 8, 2025 18:34
@tankyleo
tankyleo marked this pull request as ready for review April 9, 2025 18:25
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/channel.rs Outdated
@codecov

codecovBot commented Apr 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 99.15254% with 2 lines in your changes missing coverage. Please review.

Project coverage is 89.21%. Comparing base (ef0fcab) to head (75f81b5).
Report is 19 commits behind head on main.

Files with missing linesPatch %Lines
lightning/src/ln/chan_utils.rs98.81%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #3689 +/- ##
==========================================
+ Coverage 89.07% 89.21% +0.13% 
==========================================
Files 156 156 Lines 123507 124130 +623 Branches 123507 124130 +623 ==========================================
+ Hits 110017 110742 +725 + Misses 10801 10724 -77 + Partials 2689 2664 -25 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last real comment

Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 4 times, most recently from 1d7a1d8 to 262351cCompareApril 12, 2025 03:47

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New commit basically LTGM.

Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Probably needs another look from @wpaulino otherwise I'm happy.

Comment threadlightning/src/ln/chan_utils.rs Outdated
///
/// This is not exported to bindings users due to the generic though we likely should expose a version without
pub fn new_with_auxiliary_htlc_data<T>(commitment_number: u64, per_commitment_point: &PublicKey, to_broadcaster_value_sat: u64, to_countersignatory_value_sat: u64, feerate_per_kw: u32, htlcs_with_aux: &mut Vec<(HTLCOutputInCommitment, T)>, channel_parameters: &DirectedChannelTransactionParameters, secp_ctx: &Secp256k1<secp256k1::All>) -> CommitmentTransaction {
/// All HTLCs MUST be above the dust limit for the channel.\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray , I think? note that if you dont leave an extra line between paragraphs, rustdoc assumes you meant for them to be in the same paragraph and ignores newlines in the source.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rustdoc actually turns \ into a single newline character.
image

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, TIL, but it also makes the source unreadable :)

Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 8bc75b9 to 5252437CompareApril 15, 2025 02:10
Comment threadlightning/src/ln/channel.rs
Comment threadlightning/src/ln/chan_utils.rs Outdated
Comment threadlightning/src/ln/chan_utils.rs Outdated
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from b82a9d5 to b44f272CompareApril 15, 2025 18:45
@tankyleo
tankyleo requested a review from wpaulinoApril 15, 2025 18:45
@tankyleo
tankyleoforce-pushed the small-n-squared branch 2 times, most recently from 2f7567d to 5e67a1cCompareApril 15, 2025 19:01
wpaulino
wpaulino previously approved these changes Apr 15, 2025
TheBlueMatt
TheBlueMatt previously approved these changes Apr 16, 2025

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding the sort in the rebuild case, but gonna go ahead and land it.

let (obscured_commitment_transaction_number, txins) = Self::build_inputs(self.commitment_number, channel_parameters);

// First rebuild the htlc outputs, note that these are already sorted
let mut outputs = Self::build_htlc_outputs(keys, &self.nondust_htlcs, channel_parameters.channel_type_features());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to resort the HTLCs here before we return. The point of rebuild_transaction is to avoid trusting the pre-built commitment transaction entirely and build it from the HTLC list.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I get this done now ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have fn verify take a &mut self ? To highlight that this call will resort the HTLCs if necessary. Then fn rebuild_transaction can call build_outputs_and_htlcs just like fn new without any new memory allocations.

@tankyleotankyleoApr 16, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah looks like this upsets the validation APIs of the signer that take a &CommitmentTransaction

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we make it return a Result? Can happen in a followup but up to you if you want to do it now.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning rebuild_transaction returns an Err if the HTLCs are not sorted ? Yes

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol, didn't mean for the nit on docs to be blocking.

@tankyleo

Copy link
Copy Markdown
ContributorAuthor

Ah all good I do agree with the aesthetics.

This commit reworks how channel is told about which HTLCs were assigned
which index upon the build of a `CommitmentTransaction`.
Previously, `CommitmentTransaction` was given an exclusive reference to
channel's HTLC-source table, and `CommitmentTransaction` populated the
transaction output indices in that table via this exclusive reference.
As a result, the public API of `CommitmentTransaction` included a
generic parameter, and an exclusive reference.
We remove both of these in preparation for the upcoming `TxBuilder`
trait. This cleans up the API, and makes it more bindings-friendly.
Henceforth, channel populates the HTLC-source table via a brute-force
search of each htlc in `CommitmentTransaction`. This is an O(n^2)
operation, but n is small enough that we ignore the performance hit.
We also take this opportunity to cleanup how we build and sort the
commitment transaction outputs together with the htlc output data
in `CommitmentTransaction`. The goal is to keep the number of vector
allocations to a minimum; we now only allocate a single vector to hold
the transaction outputs, and do all the sorting in-place.
Finally, we add test coverage to assert that
`CommitmentTransaction::verify` actually checks that the HTLCs
in the `nondust_htlcs` member are actually sorted.
@tankyleo

Copy link
Copy Markdown
ContributorAuthor

@wpaulino I previously mistakenly started trusting the order of HTLCs in the nondust_htlcs member in the verify call - I correct this here.

@tankyleo
tankyleo requested a review from wpaulinoApril 16, 2025 20:01
@TheBlueMatt
TheBlueMatt merged commit 22a18a8 into lightningdevkit:mainApr 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tankyleo@ldk-reviews-bot@TheBlueMatt@wpaulino