Replace use of HolderSignedTx with HolderCommitment by wpaulino · Pull Request #3664 · lightningdevkit/rust-lightning

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino · 2025-03-12T17:41:09Z

As we move to a custom commitment transactions world, we'll no longer be able to rely on HolderSignedTx and should instead track the actual commitment transaction itself. This commit does so by introducing a new HolderCommitment struct, which we'll use as a replacement to HolderSignedTx to hold all relevant holder commitment data, including the HolderCommitmentTransaction, without data duplication.

Luckily, this is a backwards and forwards compatible change due to the OnchainTxHandler tracking the latest HolderCommitmentTransactions. In the future, we aim to remove them from the OnchainTxHandler entirely and begin storing them at the ChannelMonitor level.

Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

Depends on #3663.

ldk-reviews-bot · 2025-03-12T17:41:12Z

👋 Thanks for assigning @jkczyz as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-03-12T18:43:39Z

+ let holder_commitment = HolderCommitment::try_from((
+ current_holder_commitment_tx, &current_holder_signed_tx,
+ )).map_err(|_| DecodeError::InvalidValue)?;
+ if HolderSignedTx::from(&holder_commitment) != current_holder_signed_tx {


What's gonna happen here? Are we just gonna keep writing the legacy versions and converting, or should we have some upgrade path?

Was planning to address this in a follow-up PR that removes them from OnchainTxHandler such that we can no longer rely on getting them from there and instead store them as a new TLV in the monitor.

ldk-reviews-bot · 2025-03-12T18:43:44Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

TheBlueMatt · 2025-03-12T19:26:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

codecov · 2025-03-12T19:57:56Z

Codecov Report

Attention: Patch coverage is 89.71963% with 33 lines in your changes missing coverage. Please review.

Project coverage is 89.15%. Comparing base (030a784) to head (5d13fdc).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/chain/channelmonitor.rs	89.57%	19 Missing and 13 partials ⚠️
lightning/src/ln/channel.rs	88.88%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #3664 +/- ##
==========================================
- Coverage 89.25% 89.15% -0.11% 
==========================================
Files 155 155 Lines 119959 121059 +1100 Branches 119959 121059 +1100 ==========================================
+ Hits 107069 107925 +856 - Misses 10276 10478 +202 - Partials 2614 2656 +42

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wpaulino · 2025-03-12T22:55:53Z

This does seem like the kind of thing that might want an 👋 upgrade test 👋

Will need to wait for #3584 first

dunxen · 2025-03-17T12:16:45Z

Needs rebase.

tankyleo

Didn't make it through everything, some initial comments.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

jkczyz

Made it through everything. Nothing out of the ordinary pops out aside from what other reviewers have raised.

I'm also unclear how this helps with custom commitment transactions in particular ? Thanks for clarifying here.

Will let @wpaulino confirm, but IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

wpaulino · 2025-03-17T23:56:49Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.

Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

tankyleo · 2025-03-18T02:32:42Z

IIUC prior to this PR we were only storing parts of the commitment transaction, so anything additional in a custom one would be missing.
Correct. We were already tracking the actual commitment transaction at the OnchainTxHandler, so HolderSignedTx was redundant. The plan is to move the commitment transaction storage from OnchainTxHandler -> ChannelMonitor.

Ok! From my side here's how I've been approaching custom transactions in channel monitor. Many details I haven't worked out yet but big picture:

While storing the fully built holder commitment transactions is ok as we only store two, we will likely not store all the previous counterparty commitment transactions (even custom ones), as the storage cost would be too high.
For this problem, me and Matt have been going for a roundtrip to some signer-like object to generate the claims.
ie "hey signer ! here's a commitment tx that confirmed, what can we claim out of this?"
this could work for all the counterparty transactions (revoked and non-revoked), and I've been making it work for holder transactions too.

Experimental API currently looks like this:

/// Returns claims on a counterparty commitment transactionfngenerate_claims_from_counterparty_tx(&self,per_commitment_point:&PublicKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a counterparty's revoked commitment transactionfngenerate_claims_from_revoked_tx(&self,per_commitment_key:&SecretKey,channel_parameters:&ChannelTransactionParameters,tx:&Transaction,per_commitment_claimable_data:&Vec<(HTLCOutputInCommitment,Option<Box<HTLCSource>>)>,height:u32,secp_ctx:&Secp256k1<secp256k1::All>,) -> (Vec<PackageTemplate>,CommitmentTxCounterpartyOutputInfo);/// Returns claims on a holder's commitment transactionfngenerate_claims_from_holder_tx(&self,holder_tx:&HolderSignedTx,conf_height:u32,channel_parameters:&ChannelTransactionParameters,payment_preimages:&HashMap<PaymentHash,(PaymentPreimage,Vec<PaymentClaimDetails>)>,secp_ctx:&Secp256k1<secp256k1::All>,) -> Vec<PackageTemplate>;

What's missing:

When we force close, we want to generate claims even though our holder commit tx hasn't confirmed yet.
When we learn a preimage, we want to generate claims against already confirmed outputs.
Generate claims on htlc transactions.

Let me know if this syncs up with what you were thinking @wpaulino happy to chat here or elsewhere.

This branch here contains this API together with an implementation that passes the test suite:

https://github.com/tankyleo/rust-lightning/tree/check

jkczyz

LGTM aside from pending discussions. Please re-request a review once those are resolved.

wpaulino · 2025-03-18T17:04:31Z

@tankyleo your proposal seems fine so far. The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway. The rest is mostly unrelated to this change, so happy to chat about this work offline.

tankyleo · 2025-03-18T17:07:54Z

The only change as a result of this PR would be to the function argument in generate_claims_from_holder_tx, we would've probably wanted to provide the HolderCommitmentTransaction there to begin with anyway.

Yes agreed

The rest is mostly unrelated to this change, so happy to chat about this work offline.

Sounds good

TheBlueMatt · 2025-03-25T12:42:56Z

 Ok(LatestHolderCommitmentTXInfo {
 commitment_tx: holder_commitment_tx,
- htlc_outputs: htlcs_and_sigs,
+ htlc_outputs: dust_htlcs,


Can we rename this now that its something different?

It might still have the old meaning when we read. Would you rather change it now anyway or wait until we know for sure we'll only ever find dust_htlcs?

Maybe mostly_dust_htlcs? :). I mean we can certainly do it later, just seems easy for a bug to slip in because of a bad name.

I'd rather defer it. I don't think a bug could happen where non-dust HTLCs are included because we still have code to support that, and the current name already assumed dust HTLCs must be included.

Currently, non-dust HTLCs are duplicated across the commitment transaction itself, and the full set of HTLCs (dust & non-dust) along with their `HTLCSource` considered in the commitment transaction. As of v0.0.15, we've had support for providing non-dust HTLC sources separately such that we no longer track duplicate non-dust HTLC data, but only enabled it under testing environments. This commit enables it such that it always happens. Note that we still need to support reading `ChannelMonitorUpdateStep::LatestHolderCommitmentTXInfo` updates that did not separate the non-dust HTLC sources in case they were written in an older version and they've yet to be processed.

wpaulino · 2025-03-25T17:04:50Z

Had to rebase due to a conflict with #3606 and implemented write_as_legacy.

TheBlueMatt

Would be nice to reduce the clones in a few places still, but we can also do it in a followup.

As we move to a custom commitment transactions world, we'll no longer be able to rely on `HolderSignedTx` and should instead track the actual commitment transaction itself. This commit does so by introducing a new `HolderCommitment` struct, which we'll use as a replacement to `HolderSignedTx` to hold all relevant holder commitment data, including the `HolderCommitmentTransaction`, without data duplication. Luckily, this is a backwards and forwards compatible change due to the `OnchainTxHandler` tracking the latest `HolderCommitmentTransaction`s. In the future, we aim to remove them from the `OnchainTxHandler` entirely and begin storing them at the `ChannelMonitor` level. Aside from how the data is represented internally, there shouldn't be any functional changes within this patch.

ldk-reviews-bot · 2025-03-31T00:00:53Z

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

wpaulino added the weekly goal Someone wants to land this this week label Mar 12, 2025

wpaulino requested a review from TheBlueMatt March 12, 2025 17:41

TheBlueMatt reviewed Mar 12, 2025
View reviewed changes

wpaulino force-pushed the deprecate-holder-signed-tx branch from 2884bad to fdc1706Compare March 12, 2025 23:04

wpaulino force-pushed the deprecate-holder-signed-tx branch from fdc1706 to 73f82fbCompare March 17, 2025 16:53

wpaulino requested review from TheBlueMatt and jkczyz March 17, 2025 17:03

tankyleo reviewed Mar 17, 2025
View reviewed changes

jkczyz reviewed Mar 17, 2025
View reviewed changes

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs Outdated

wpaulino force-pushed the deprecate-holder-signed-tx branch from 73f82fb to 24dffdbCompare March 17, 2025 23:55

jkczyz reviewed Mar 18, 2025
View reviewed changes

wpaulino requested a review from jkczyz March 18, 2025 17:01

jkczyz previously approved these changes Mar 18, 2025
View reviewed changes

TheBlueMatt removed their request for review March 18, 2025 19:13

wpaulino requested a review from TheBlueMatt March 24, 2025 22:37

TheBlueMatt reviewed Mar 25, 2025
View reviewed changes

wpaulino dismissed jkczyz’s stale review via 41d9463March 25, 2025 17:03

wpaulino force-pushed the deprecate-holder-signed-tx branch from 24dffdb to 41d9463Compare March 25, 2025 17:03

wpaulino force-pushed the deprecate-holder-signed-tx branch from 41d9463 to 7f626c8Compare March 25, 2025 17:03

wpaulino requested a review from TheBlueMatt March 25, 2025 17:05

TheBlueMatt reviewed Mar 25, 2025
View reviewed changes

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning/src/chain/channelmonitor.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated

wpaulino force-pushed the deprecate-holder-signed-tx branch from 7f626c8 to a891a04Compare March 25, 2025 22:31

wpaulino requested a review from TheBlueMatt March 25, 2025 22:32

TheBlueMatt reviewed Mar 26, 2025
View reviewed changes

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning/src/chain/channelmonitor.rs

wpaulino force-pushed the deprecate-holder-signed-tx branch from a891a04 to e8ec314Compare March 26, 2025 22:58

wpaulino requested a review from TheBlueMatt March 26, 2025 23:00

TheBlueMatt removed their request for review March 27, 2025 13:27

wpaulino force-pushed the deprecate-holder-signed-tx branch from e8ec314 to 80ac853Compare March 27, 2025 15:56

wpaulino requested a review from TheBlueMatt March 27, 2025 15:56

TheBlueMatt previously approved these changes Mar 27, 2025
View reviewed changes

Comment threadlightning/src/chain/channelmonitor.rs Outdated

wpaulino dismissed TheBlueMatt’s stale review via 92df40eMarch 27, 2025 19:51

wpaulino force-pushed the deprecate-holder-signed-tx branch 2 times, most recently from 92df40e to b8007c9Compare March 28, 2025 17:29

wpaulino requested review from TheBlueMatt and jkczyz March 28, 2025 17:29

TheBlueMatt previously approved these changes Mar 28, 2025
View reviewed changes

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning/src/chain/channelmonitor.rs Outdated

wpaulino dismissed TheBlueMatt’s stale review via d279b82March 28, 2025 21:38

wpaulino force-pushed the deprecate-holder-signed-tx branch from b8007c9 to d279b82Compare March 28, 2025 21:38

wpaulino force-pushed the deprecate-holder-signed-tx branch from d279b82 to 5d13fdcCompare March 28, 2025 21:41

TheBlueMatt approved these changes Mar 29, 2025
View reviewed changes

jkczyz approved these changes Mar 31, 2025
View reviewed changes

wpaulino merged commit 7af3c8f into lightningdevkit:mainMar 31, 2025

wpaulino deleted the deprecate-holder-signed-tx branch March 31, 2025 22:10

wpaulino mentioned this pull request Apr 15, 2025
Produce a single monitor update per commitment number #3738
Closed

jkczyz mentioned this pull request Jun 21, 2025
Dual-funded channels and Splicing Project Tracking #1621
Closed
7 tasks