Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Expose onchain transactions in store by tnull · Pull Request #432 · lightningdevkit/ldk-node · GitHub
Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Expose onchain transactions in store by tnull · Pull Request #432 · lightningdevkit/ldk-node · GitHub
Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Expose onchain transactions in store by tnull · Pull Request #432 · lightningdevkit/ldk-node · GitHub
Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Expose onchain transactions in store by tnull · Pull Request #432 · lightningdevkit/ldk-node · GitHub
Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Expose onchain transactions in store by tnull · Pull Request #432 · lightningdevkit/ldk-node · GitHub
Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Expose onchain transactions in store by tnull · Pull Request #432 · lightningdevkit/ldk-node · GitHub
Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Expose onchain transactions in store by tnull · Pull Request #432 · lightningdevkit/ldk-node · GitHub
Skip to content

Expose onchain transactions in store - #432

Merged
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store
Jan 30, 2025
Merged

Expose onchain transactions in store#432
tnull merged 6 commits into
lightningdevkit:mainfrom
tnull:2025-01-include-onchain-transactions-in-store

Conversation

@tnull

@tnulltnull commented Jan 16, 2025

Copy link
Copy Markdown
Collaborator

Closes#67

Previously, onchain-transactions where not tracked in our PaymentStore.
Now, that the upgrade to BDK 1.0 is behind us, we take a stab at finally exposing them in our interface via PaymentStore.

In the future we're looking to add dedicated PaymentKinds for channel fundings/channel closes (see #447), but for now all are exposed under the Onchain umbrella kind.

@tnull
tnull marked this pull request as draft January 16, 2025 13:28
@tnulltnull added this to the 0.5 milestone Jan 16, 2025
@tnulltnull mentioned this pull request Jan 23, 2025
10 tasks
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from b959830 to 306a78eCompareJanuary 27, 2025 15:45
@tnulltnull changed the title Expose onchain transactions in store and eventsExpose onchain transactions in storeJan 27, 2025
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

As we currently don't have a good wait to discern closing transactions, I now split out the event emission to #448 which will land at a later date. Going ahead and undrafting this.

@tnull
tnull marked this pull request as ready for review January 27, 2025 15:57
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 585e14f to 6437515CompareJanuary 27, 2025 16:45
@tnull
tnull requested a review from jkczyzJanuary 27, 2025 16:46
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 6437515 to 403be03CompareJanuary 27, 2025 17:03
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs
Comment threadsrc/payment/store.rs Outdated
Comment threadsrc/wallet/mod.rs
Comment on lines +177 to +188
// TODO: It would be great to introduce additional variants for
// `ChannelFunding` and `ChannelClosing`. For the former, we could just
// take a reference to `ChannelManager` here and check against
// `list_channels`. But for the latter the best approach is much less
// clear: for force-closes/HTLC spends we should be good querying
// `OutputSweeper::tracked_spendable_outputs`, but regular channel closes
// (i.e., `SpendableOutputDescriptor::StaticOutput` variants) are directly
// spent to a wallet address. The only solution I can come up with is to
// create and persist a list of 'static pending outputs' that we could use
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this gets more complicated with splicing in/out or more exotic transactions paying us and opening a channel, for instance.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good point, now opened the more lightningdevkit/rust-lightning#3566 to supersede lightningdevkit/rust-lightning#3548

Would be great if an API allowing to query the type of a given transaction (Txid) could be added as part of the splicing work.

Comment threadsrc/wallet/mod.rs
Comment on lines +190 to +199
let (direction, amount_msat) = if sent > received {
let direction = PaymentDirection::Outbound;
let amount_msat = Some(sent.to_sat().saturating_sub(received.to_sat()) * 1000);
(direction, amount_msat)
} else {
let direction = PaymentDirection::Inbound;
let amount_msat = Some(received.to_sat().saturating_sub(sent.to_sat()) * 1000);
(direction, amount_msat)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider these two separate payments? I guess we could't use the txid as the payment id, though.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhh, given that (IIUC) received includes the change, I'd rather not add each change output as an inbound payment?

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Say the transaction was re-orged out and was then RBF'ed. Is that possible? Would we have payment stuck in pending?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a very valid concern. Tbh. I'm not entirely sure how to handle that: for one, the RBF'd transaction is technically pending as there could be another reorg that makes it part of the best chain again. So essentially any signed & broadcasted transaction remains 'pending' forever.

I think the only way I can come up with to handle this would be to never add Pending transactions but jump to Succeeded directly once they reach ANTI_REORG_DELAY. Or we just keep the current approach and add docs noting that transactions in pending might never succeed for one reason or another?

I think at least for BDK's own RBF's we could see if we find a better solution when we do #367.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

@tnulltnullJan 30, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, but that would require us to keep track and persist a list of all the UTXOs, write an adapter around lightning-transaction-sync's Filter implementation to also register spends for them, and then implement Confirm on Wallet, just to be able to remove any Pending entries that have been replaced.

For now that seems like a lot of overhead to just account for this rare edgecase (reorg + RBF)? But I wouldn't completely rule it out to do something like it in if we find we need to do additional RBF tracking anyways as part of #367.

Comment threadsrc/wallet/mod.rs
// here to determine the `PaymentKind`, but that's not really satisfactory, so
// we're punting on it until we can come up with a better solution.
let kind = crate::payment::PaymentKind::Onchain { txid, status: confirmation_status };
let (sent, received) = locked_wallet.sent_and_received(&wtx.tx_node.tx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is change abstracted away from this API?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the docs state:

 /// This method returns a tuple `(sent, received)`. Sent is the sum of the txin amounts
/// that spend from previous txouts tracked by this wallet. Received is the summation
/// of this tx's outputs that send to script pubkeys tracked by this wallet.

so IIUC received would include the change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't the change be more than the amount sent, making it look like an inbound payment when it was really an outbound? Or am I misunderstanding?

IIUC, any change amount would be accounted for both in sent and received, so you can essentially ignore it in terms of comparisons (think: subtract on both sides).

Say you send a payment of 10 sats, and add a txin spending an wallet output with 100 sats. This would result in two txouts, one to the recipient with 10 sats, one change back to a wallet address with 85sats or so, leaving 5 for mining fees. IIUC, this would result in sent = 100, received = 85, resulting in sent > received => PaymentDirection::Outbound.

@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch 2 times, most recently from 1d05f9c to 993bf0bCompareJanuary 28, 2025 10:50

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash

Comment threadsrc/wallet/mod.rs
Comment on lines +160 to +166
let payment_status = if cur_height >= confirmation_height + ANTI_REORG_DELAY - 1
{
PaymentStatus::Succeeded
} else {
PaymentStatus::Pending
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly detect that the UTXO has been spent and remove the pending payment?

Here, we move updating fields via `PaymentDetailsUpdate` to
`PaymentDetails::update`. This allows us to only update the fields that
changed, keeping track *whether* something changed, and only updating
the timestamp and persisting the entry *if* something changed.
This is a nice improvement in general (as we want to reduce persist
calls anyways), but we'll also use this for batch updates in the next
commits.
We implement an inserting/updating method that will only persist entries that
have been changed.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from 993bf0b to da45da3CompareJanuary 30, 2025 12:56
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Rebased to resolve minor conflicts

Previously, `PaymentKind::Onchain` was simply a placeholder entry we
never actually used. Here, we extend it to include fields that are
actually useful.
We update the payment store whenever syncing the wallet state finished.
@tnull
tnullforce-pushed the 2025-01-include-onchain-transactions-in-store branch from da45da3 to 0710434CompareJanuary 30, 2025 16:03
@tnull

Copy link
Copy Markdown
CollaboratorAuthor

Squashed the fixup without further changes.

@tnull
tnull merged commit aaf74cc into lightningdevkit:mainJan 30, 2025
@danielnordh

Copy link
Copy Markdown

Yes, thank you for getting this done!!

@reezreez mentioned this pull request Feb 3, 2025
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose on-chain transactions

3 participants

@tnull@danielnordh@jkczyz