Remove KeysInterface by arik-so · Pull Request #1930 · lightningdevkit/rust-lightning

arik-so · 2022-12-20T23:38:49Z

No description provided.

codecov-commenter · 2022-12-21T01:14:55Z

Codecov Report

Base: 90.73% // Head: 90.76% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (22d7a0b) compared to base (5221e4a).
Patch coverage: 97.60% of modified lines in pull request are covered.

❗ Current head 22d7a0b differs from pull request most recent head 49c1f30. Consider uploading reports for the commit 49c1f30 to get more accurate results

Additional details and impacted files

@@ Coverage Diff @@## main #1930 +/- ##
==========================================
+ Coverage 90.73% 90.76% +0.02% 
==========================================
Files 97 96 -1 Lines 50539 50088 -451 Branches 50539 50088 -451 ==========================================
- Hits 45859 45461 -398 + Misses 4680 4627 -53

Impacted Files	Coverage Δ
lightning-block-sync/src/init.rs	`91.11% <ø> (ø)`
lightning/src/chain/keysinterface.rs	`83.78% <ø> (-0.04%)`	⬇️
lightning/src/ln/chan_utils.rs	`93.56% <ø> (ø)`
lightning/src/ln/onion_route_tests.rs	`97.64% <ø> (-0.01%)`	⬇️
lightning/src/ln/payment_tests.rs	`98.73% <ø> (+<0.01%)`	⬆️
lightning/src/ln/peer_handler.rs	`55.82% <ø> (ø)`
lightning/src/ln/priv_short_conf_tests.rs	`96.54% <ø> (ø)`
lightning/src/ln/reorg_tests.rs	`100.00% <ø> (ø)`
lightning/src/ln/shutdown_tests.rs	`96.51% <ø> (ø)`
lightning/src/routing/router.rs	`90.90% <ø> (+0.03%)`	⬆️
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wpaulino · 2022-12-21T21:08:10Z

 	CW::Target: 'static + chain::Watch<<K::Target as SignerProvider>::Signer>,
 	T::Target: 'static + BroadcasterInterface,
-	K::Target: 'static + KeysInterface,
+	K::Target: 'static + EntropySource + NodeSigner + SignerProvider,


I imagine we can probably start to split each of these out into their own generic constraint (yay more generics). EntropySource seems like a good candidate since it's somewhat standalone. NodeSigner and SignerProvider really depends on the underlying deployment, since they will likely be backed by the same HSM/secure element.

Yea, if we're splitting the traits I have to assume we expect people to have different structs that implement different parts of the traits (otherwise why are we doing this to begin with :) ), in which case we need to break up the generics as well.

I agree we should split things up, but I would lean against doing all the splitting in this PR, maybe with the exception of some very shallow-level API examples, like BackgroundProcessor. BackgroundProcessor definitely lacks generics.

I don't think it will be less effort to do it in one PR? As-is is just more diff and intermediate state, it would be less total diff to review doing it all at once.

Well, one benefit of splitting it up into two is that it would allow @wpaulino to open his node_secret removal PR against the commit that's currently in this one, while I work on the follow-up in parallel. But I'll work on the parameter split tonight; maybe I can get it in fast enough and it's less trouble than I'm anticipating.

I don't think that would conflict, now that we split the traits and moved those functions around. This PR doesn't really heavily touch the keysinterface.rs file.

arik-so · 2022-12-22T21:23:33Z

Wilmer just made the point that instead of removing KeysInterface entirely, we could simply make it be comprised of NodeSigner and SignerProvider, removing its EntropySource component.

TheBlueMatt · 2022-12-22T22:05:42Z

I'm still not sure why its worth doing that as a separate PR, but, sure, if you want to do that to make forward progress, sounds fine. We'll ultimately need to do it for all of them anyway.

TheBlueMatt · 2023-01-03T17:00:34Z

This commit separation makes the PR really hard to review (ie generates more total lines of diff to look at) - can you instead break it up as individual commits which change different structs to use individual traits (including the lifetime changes) and then only as the very last commit remove the (at that point unused) KeysInterface trait? That should lead to a series of relatively small commits.

arik-so · 2023-01-05T20:34:53Z

I'll try

TheBlueMatt · 2023-01-05T22:13:15Z

Needs rebase after #1812, sorry. Should be mechanical but may be a bit annoying.

arik-so · 2023-01-11T22:05:26Z

There are only like two files that are letting their KeysInterface arguments be split up atomically, inbound_payment.rs and blinded_path.rs. Everything else percolates down to the Channel and ChannelManager struct, resulting in an all-or-nothing scenario.

TheBlueMatt · 2023-01-12T19:28:29Z

Can't OnionMessenger go separately as well? Also probably ChannelMonitor, no?

arik-so · 2023-01-12T19:41:33Z

Funny you say that, because those two are exactly the types I've been trying to make separate. But no, they both have dependency chains that suck in all of Channel and ChannelManager.

…ysInterface.

TheBlueMatt · 2023-01-13T19:23:41Z

If you remove the KeysInterface last, rather than first, I think it should work? Which dependency chain are you referring to?

arik-so · 2023-01-13T19:27:15Z

I'm talking about splitting up the keys_interface argument into implementations of its comprising traits. Once you have two or three arguments of different traits, that dependency chain sneaks through pretty much everything. That is unrelated to removing KeysInterface.

wpaulino · 2023-01-13T22:38:37Z

 	pub chain_monitor: &'b test_utils::TestChainMonitor<'c>,
 	pub keys_manager: &'b test_utils::TestKeysInterface,
-	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,
+	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,


Pre-existing, but we really need a type alias for this test ChannelManager...

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

arik-so · 2022-12-20T23:38:49Z

No description provided.

codecov-commenter · 2022-12-21T01:14:55Z

Codecov Report

Base: 90.73% // Head: 90.76% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (22d7a0b) compared to base (5221e4a).
Patch coverage: 97.60% of modified lines in pull request are covered.

❗ Current head 22d7a0b differs from pull request most recent head 49c1f30. Consider uploading reports for the commit 49c1f30 to get more accurate results

Additional details and impacted files

@@ Coverage Diff @@## main #1930 +/- ##
==========================================
+ Coverage 90.73% 90.76% +0.02% 
==========================================
Files 97 96 -1 Lines 50539 50088 -451 Branches 50539 50088 -451 ==========================================
- Hits 45859 45461 -398 + Misses 4680 4627 -53

Impacted Files	Coverage Δ
lightning-block-sync/src/init.rs	`91.11% <ø> (ø)`
lightning/src/chain/keysinterface.rs	`83.78% <ø> (-0.04%)`	⬇️
lightning/src/ln/chan_utils.rs	`93.56% <ø> (ø)`
lightning/src/ln/onion_route_tests.rs	`97.64% <ø> (-0.01%)`	⬇️
lightning/src/ln/payment_tests.rs	`98.73% <ø> (+<0.01%)`	⬆️
lightning/src/ln/peer_handler.rs	`55.82% <ø> (ø)`
lightning/src/ln/priv_short_conf_tests.rs	`96.54% <ø> (ø)`
lightning/src/ln/reorg_tests.rs	`100.00% <ø> (ø)`
lightning/src/ln/shutdown_tests.rs	`96.51% <ø> (ø)`
lightning/src/routing/router.rs	`90.90% <ø> (+0.03%)`	⬆️
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wpaulino · 2022-12-21T21:08:10Z

 	CW::Target: 'static + chain::Watch<<K::Target as SignerProvider>::Signer>,
 	T::Target: 'static + BroadcasterInterface,
-	K::Target: 'static + KeysInterface,
+	K::Target: 'static + EntropySource + NodeSigner + SignerProvider,


I imagine we can probably start to split each of these out into their own generic constraint (yay more generics). EntropySource seems like a good candidate since it's somewhat standalone. NodeSigner and SignerProvider really depends on the underlying deployment, since they will likely be backed by the same HSM/secure element.

Yea, if we're splitting the traits I have to assume we expect people to have different structs that implement different parts of the traits (otherwise why are we doing this to begin with :) ), in which case we need to break up the generics as well.

I agree we should split things up, but I would lean against doing all the splitting in this PR, maybe with the exception of some very shallow-level API examples, like BackgroundProcessor. BackgroundProcessor definitely lacks generics.

I don't think it will be less effort to do it in one PR? As-is is just more diff and intermediate state, it would be less total diff to review doing it all at once.

Well, one benefit of splitting it up into two is that it would allow @wpaulino to open his node_secret removal PR against the commit that's currently in this one, while I work on the follow-up in parallel. But I'll work on the parameter split tonight; maybe I can get it in fast enough and it's less trouble than I'm anticipating.

I don't think that would conflict, now that we split the traits and moved those functions around. This PR doesn't really heavily touch the keysinterface.rs file.

arik-so · 2022-12-22T21:23:33Z

Wilmer just made the point that instead of removing KeysInterface entirely, we could simply make it be comprised of NodeSigner and SignerProvider, removing its EntropySource component.

TheBlueMatt · 2022-12-22T22:05:42Z

I'm still not sure why its worth doing that as a separate PR, but, sure, if you want to do that to make forward progress, sounds fine. We'll ultimately need to do it for all of them anyway.

TheBlueMatt · 2023-01-03T17:00:34Z

This commit separation makes the PR really hard to review (ie generates more total lines of diff to look at) - can you instead break it up as individual commits which change different structs to use individual traits (including the lifetime changes) and then only as the very last commit remove the (at that point unused) KeysInterface trait? That should lead to a series of relatively small commits.

arik-so · 2023-01-05T20:34:53Z

I'll try

TheBlueMatt · 2023-01-05T22:13:15Z

Needs rebase after #1812, sorry. Should be mechanical but may be a bit annoying.

arik-so · 2023-01-11T22:05:26Z

There are only like two files that are letting their KeysInterface arguments be split up atomically, inbound_payment.rs and blinded_path.rs. Everything else percolates down to the Channel and ChannelManager struct, resulting in an all-or-nothing scenario.

TheBlueMatt · 2023-01-12T19:28:29Z

Can't OnionMessenger go separately as well? Also probably ChannelMonitor, no?

arik-so · 2023-01-12T19:41:33Z

Funny you say that, because those two are exactly the types I've been trying to make separate. But no, they both have dependency chains that suck in all of Channel and ChannelManager.

…ysInterface.

TheBlueMatt · 2023-01-13T19:23:41Z

If you remove the KeysInterface last, rather than first, I think it should work? Which dependency chain are you referring to?

arik-so · 2023-01-13T19:27:15Z

I'm talking about splitting up the keys_interface argument into implementations of its comprising traits. Once you have two or three arguments of different traits, that dependency chain sneaks through pretty much everything. That is unrelated to removing KeysInterface.

wpaulino · 2023-01-13T22:38:37Z

 	pub chain_monitor: &'b test_utils::TestChainMonitor<'c>,
 	pub keys_manager: &'b test_utils::TestKeysInterface,
-	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,
+	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,


Pre-existing, but we really need a type alias for this test ChannelManager...

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

arik-so · 2022-12-20T23:38:49Z

No description provided.

codecov-commenter · 2022-12-21T01:14:55Z

Codecov Report

Base: 90.73% // Head: 90.76% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (22d7a0b) compared to base (5221e4a).
Patch coverage: 97.60% of modified lines in pull request are covered.

❗ Current head 22d7a0b differs from pull request most recent head 49c1f30. Consider uploading reports for the commit 49c1f30 to get more accurate results

Additional details and impacted files

@@ Coverage Diff @@## main #1930 +/- ##
==========================================
+ Coverage 90.73% 90.76% +0.02% 
==========================================
Files 97 96 -1 Lines 50539 50088 -451 Branches 50539 50088 -451 ==========================================
- Hits 45859 45461 -398 + Misses 4680 4627 -53

Impacted Files	Coverage Δ
lightning-block-sync/src/init.rs	`91.11% <ø> (ø)`
lightning/src/chain/keysinterface.rs	`83.78% <ø> (-0.04%)`	⬇️
lightning/src/ln/chan_utils.rs	`93.56% <ø> (ø)`
lightning/src/ln/onion_route_tests.rs	`97.64% <ø> (-0.01%)`	⬇️
lightning/src/ln/payment_tests.rs	`98.73% <ø> (+<0.01%)`	⬆️
lightning/src/ln/peer_handler.rs	`55.82% <ø> (ø)`
lightning/src/ln/priv_short_conf_tests.rs	`96.54% <ø> (ø)`
lightning/src/ln/reorg_tests.rs	`100.00% <ø> (ø)`
lightning/src/ln/shutdown_tests.rs	`96.51% <ø> (ø)`
lightning/src/routing/router.rs	`90.90% <ø> (+0.03%)`	⬆️
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wpaulino · 2022-12-21T21:08:10Z

 	CW::Target: 'static + chain::Watch<<K::Target as SignerProvider>::Signer>,
 	T::Target: 'static + BroadcasterInterface,
-	K::Target: 'static + KeysInterface,
+	K::Target: 'static + EntropySource + NodeSigner + SignerProvider,


I imagine we can probably start to split each of these out into their own generic constraint (yay more generics). EntropySource seems like a good candidate since it's somewhat standalone. NodeSigner and SignerProvider really depends on the underlying deployment, since they will likely be backed by the same HSM/secure element.

Yea, if we're splitting the traits I have to assume we expect people to have different structs that implement different parts of the traits (otherwise why are we doing this to begin with :) ), in which case we need to break up the generics as well.

I agree we should split things up, but I would lean against doing all the splitting in this PR, maybe with the exception of some very shallow-level API examples, like BackgroundProcessor. BackgroundProcessor definitely lacks generics.

I don't think it will be less effort to do it in one PR? As-is is just more diff and intermediate state, it would be less total diff to review doing it all at once.

Well, one benefit of splitting it up into two is that it would allow @wpaulino to open his node_secret removal PR against the commit that's currently in this one, while I work on the follow-up in parallel. But I'll work on the parameter split tonight; maybe I can get it in fast enough and it's less trouble than I'm anticipating.

I don't think that would conflict, now that we split the traits and moved those functions around. This PR doesn't really heavily touch the keysinterface.rs file.

arik-so · 2022-12-22T21:23:33Z

Wilmer just made the point that instead of removing KeysInterface entirely, we could simply make it be comprised of NodeSigner and SignerProvider, removing its EntropySource component.

TheBlueMatt · 2022-12-22T22:05:42Z

I'm still not sure why its worth doing that as a separate PR, but, sure, if you want to do that to make forward progress, sounds fine. We'll ultimately need to do it for all of them anyway.

TheBlueMatt · 2023-01-03T17:00:34Z

This commit separation makes the PR really hard to review (ie generates more total lines of diff to look at) - can you instead break it up as individual commits which change different structs to use individual traits (including the lifetime changes) and then only as the very last commit remove the (at that point unused) KeysInterface trait? That should lead to a series of relatively small commits.

arik-so · 2023-01-05T20:34:53Z

I'll try

TheBlueMatt · 2023-01-05T22:13:15Z

Needs rebase after #1812, sorry. Should be mechanical but may be a bit annoying.

arik-so · 2023-01-11T22:05:26Z

There are only like two files that are letting their KeysInterface arguments be split up atomically, inbound_payment.rs and blinded_path.rs. Everything else percolates down to the Channel and ChannelManager struct, resulting in an all-or-nothing scenario.

TheBlueMatt · 2023-01-12T19:28:29Z

Can't OnionMessenger go separately as well? Also probably ChannelMonitor, no?

arik-so · 2023-01-12T19:41:33Z

Funny you say that, because those two are exactly the types I've been trying to make separate. But no, they both have dependency chains that suck in all of Channel and ChannelManager.

…ysInterface.

TheBlueMatt · 2023-01-13T19:23:41Z

If you remove the KeysInterface last, rather than first, I think it should work? Which dependency chain are you referring to?

arik-so · 2023-01-13T19:27:15Z

I'm talking about splitting up the keys_interface argument into implementations of its comprising traits. Once you have two or three arguments of different traits, that dependency chain sneaks through pretty much everything. That is unrelated to removing KeysInterface.

wpaulino · 2023-01-13T22:38:37Z

 	pub chain_monitor: &'b test_utils::TestChainMonitor<'c>,
 	pub keys_manager: &'b test_utils::TestKeysInterface,
-	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,
+	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,


Pre-existing, but we really need a type alias for this test ChannelManager...

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

arik-so · 2022-12-20T23:38:49Z

No description provided.

codecov-commenter · 2022-12-21T01:14:55Z

Codecov Report

Base: 90.73% // Head: 90.76% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (22d7a0b) compared to base (5221e4a).
Patch coverage: 97.60% of modified lines in pull request are covered.

❗ Current head 22d7a0b differs from pull request most recent head 49c1f30. Consider uploading reports for the commit 49c1f30 to get more accurate results

Additional details and impacted files

@@ Coverage Diff @@## main #1930 +/- ##
==========================================
+ Coverage 90.73% 90.76% +0.02% 
==========================================
Files 97 96 -1 Lines 50539 50088 -451 Branches 50539 50088 -451 ==========================================
- Hits 45859 45461 -398 + Misses 4680 4627 -53

Impacted Files	Coverage Δ
lightning-block-sync/src/init.rs	`91.11% <ø> (ø)`
lightning/src/chain/keysinterface.rs	`83.78% <ø> (-0.04%)`	⬇️
lightning/src/ln/chan_utils.rs	`93.56% <ø> (ø)`
lightning/src/ln/onion_route_tests.rs	`97.64% <ø> (-0.01%)`	⬇️
lightning/src/ln/payment_tests.rs	`98.73% <ø> (+<0.01%)`	⬆️
lightning/src/ln/peer_handler.rs	`55.82% <ø> (ø)`
lightning/src/ln/priv_short_conf_tests.rs	`96.54% <ø> (ø)`
lightning/src/ln/reorg_tests.rs	`100.00% <ø> (ø)`
lightning/src/ln/shutdown_tests.rs	`96.51% <ø> (ø)`
lightning/src/routing/router.rs	`90.90% <ø> (+0.03%)`	⬆️
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wpaulino · 2022-12-21T21:08:10Z

 	CW::Target: 'static + chain::Watch<<K::Target as SignerProvider>::Signer>,
 	T::Target: 'static + BroadcasterInterface,
-	K::Target: 'static + KeysInterface,
+	K::Target: 'static + EntropySource + NodeSigner + SignerProvider,


I imagine we can probably start to split each of these out into their own generic constraint (yay more generics). EntropySource seems like a good candidate since it's somewhat standalone. NodeSigner and SignerProvider really depends on the underlying deployment, since they will likely be backed by the same HSM/secure element.

Yea, if we're splitting the traits I have to assume we expect people to have different structs that implement different parts of the traits (otherwise why are we doing this to begin with :) ), in which case we need to break up the generics as well.

I agree we should split things up, but I would lean against doing all the splitting in this PR, maybe with the exception of some very shallow-level API examples, like BackgroundProcessor. BackgroundProcessor definitely lacks generics.

I don't think it will be less effort to do it in one PR? As-is is just more diff and intermediate state, it would be less total diff to review doing it all at once.

Well, one benefit of splitting it up into two is that it would allow @wpaulino to open his node_secret removal PR against the commit that's currently in this one, while I work on the follow-up in parallel. But I'll work on the parameter split tonight; maybe I can get it in fast enough and it's less trouble than I'm anticipating.

I don't think that would conflict, now that we split the traits and moved those functions around. This PR doesn't really heavily touch the keysinterface.rs file.

arik-so · 2022-12-22T21:23:33Z

Wilmer just made the point that instead of removing KeysInterface entirely, we could simply make it be comprised of NodeSigner and SignerProvider, removing its EntropySource component.

TheBlueMatt · 2022-12-22T22:05:42Z

I'm still not sure why its worth doing that as a separate PR, but, sure, if you want to do that to make forward progress, sounds fine. We'll ultimately need to do it for all of them anyway.

TheBlueMatt · 2023-01-03T17:00:34Z

This commit separation makes the PR really hard to review (ie generates more total lines of diff to look at) - can you instead break it up as individual commits which change different structs to use individual traits (including the lifetime changes) and then only as the very last commit remove the (at that point unused) KeysInterface trait? That should lead to a series of relatively small commits.

arik-so · 2023-01-05T20:34:53Z

I'll try

TheBlueMatt · 2023-01-05T22:13:15Z

Needs rebase after #1812, sorry. Should be mechanical but may be a bit annoying.

arik-so · 2023-01-11T22:05:26Z

There are only like two files that are letting their KeysInterface arguments be split up atomically, inbound_payment.rs and blinded_path.rs. Everything else percolates down to the Channel and ChannelManager struct, resulting in an all-or-nothing scenario.

TheBlueMatt · 2023-01-12T19:28:29Z

Can't OnionMessenger go separately as well? Also probably ChannelMonitor, no?

arik-so · 2023-01-12T19:41:33Z

Funny you say that, because those two are exactly the types I've been trying to make separate. But no, they both have dependency chains that suck in all of Channel and ChannelManager.

…ysInterface.

TheBlueMatt · 2023-01-13T19:23:41Z

If you remove the KeysInterface last, rather than first, I think it should work? Which dependency chain are you referring to?

arik-so · 2023-01-13T19:27:15Z

I'm talking about splitting up the keys_interface argument into implementations of its comprising traits. Once you have two or three arguments of different traits, that dependency chain sneaks through pretty much everything. That is unrelated to removing KeysInterface.

wpaulino · 2023-01-13T22:38:37Z

 	pub chain_monitor: &'b test_utils::TestChainMonitor<'c>,
 	pub keys_manager: &'b test_utils::TestKeysInterface,
-	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,
+	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,


Pre-existing, but we really need a type alias for this test ChannelManager...

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

arik-so · 2022-12-20T23:38:49Z

No description provided.

codecov-commenter · 2022-12-21T01:14:55Z

Codecov Report

Base: 90.73% // Head: 90.76% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (22d7a0b) compared to base (5221e4a).
Patch coverage: 97.60% of modified lines in pull request are covered.

❗ Current head 22d7a0b differs from pull request most recent head 49c1f30. Consider uploading reports for the commit 49c1f30 to get more accurate results

Additional details and impacted files

@@ Coverage Diff @@## main #1930 +/- ##
==========================================
+ Coverage 90.73% 90.76% +0.02% 
==========================================
Files 97 96 -1 Lines 50539 50088 -451 Branches 50539 50088 -451 ==========================================
- Hits 45859 45461 -398 + Misses 4680 4627 -53

Impacted Files	Coverage Δ
lightning-block-sync/src/init.rs	`91.11% <ø> (ø)`
lightning/src/chain/keysinterface.rs	`83.78% <ø> (-0.04%)`	⬇️
lightning/src/ln/chan_utils.rs	`93.56% <ø> (ø)`
lightning/src/ln/onion_route_tests.rs	`97.64% <ø> (-0.01%)`	⬇️
lightning/src/ln/payment_tests.rs	`98.73% <ø> (+<0.01%)`	⬆️
lightning/src/ln/peer_handler.rs	`55.82% <ø> (ø)`
lightning/src/ln/priv_short_conf_tests.rs	`96.54% <ø> (ø)`
lightning/src/ln/reorg_tests.rs	`100.00% <ø> (ø)`
lightning/src/ln/shutdown_tests.rs	`96.51% <ø> (ø)`
lightning/src/routing/router.rs	`90.90% <ø> (+0.03%)`	⬆️
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wpaulino · 2022-12-21T21:08:10Z

 	CW::Target: 'static + chain::Watch<<K::Target as SignerProvider>::Signer>,
 	T::Target: 'static + BroadcasterInterface,
-	K::Target: 'static + KeysInterface,
+	K::Target: 'static + EntropySource + NodeSigner + SignerProvider,


I imagine we can probably start to split each of these out into their own generic constraint (yay more generics). EntropySource seems like a good candidate since it's somewhat standalone. NodeSigner and SignerProvider really depends on the underlying deployment, since they will likely be backed by the same HSM/secure element.

Yea, if we're splitting the traits I have to assume we expect people to have different structs that implement different parts of the traits (otherwise why are we doing this to begin with :) ), in which case we need to break up the generics as well.

I agree we should split things up, but I would lean against doing all the splitting in this PR, maybe with the exception of some very shallow-level API examples, like BackgroundProcessor. BackgroundProcessor definitely lacks generics.

I don't think it will be less effort to do it in one PR? As-is is just more diff and intermediate state, it would be less total diff to review doing it all at once.

Well, one benefit of splitting it up into two is that it would allow @wpaulino to open his node_secret removal PR against the commit that's currently in this one, while I work on the follow-up in parallel. But I'll work on the parameter split tonight; maybe I can get it in fast enough and it's less trouble than I'm anticipating.

I don't think that would conflict, now that we split the traits and moved those functions around. This PR doesn't really heavily touch the keysinterface.rs file.

arik-so · 2022-12-22T21:23:33Z

Wilmer just made the point that instead of removing KeysInterface entirely, we could simply make it be comprised of NodeSigner and SignerProvider, removing its EntropySource component.

TheBlueMatt · 2022-12-22T22:05:42Z

I'm still not sure why its worth doing that as a separate PR, but, sure, if you want to do that to make forward progress, sounds fine. We'll ultimately need to do it for all of them anyway.

TheBlueMatt · 2023-01-03T17:00:34Z

This commit separation makes the PR really hard to review (ie generates more total lines of diff to look at) - can you instead break it up as individual commits which change different structs to use individual traits (including the lifetime changes) and then only as the very last commit remove the (at that point unused) KeysInterface trait? That should lead to a series of relatively small commits.

arik-so · 2023-01-05T20:34:53Z

I'll try

TheBlueMatt · 2023-01-05T22:13:15Z

Needs rebase after #1812, sorry. Should be mechanical but may be a bit annoying.

arik-so · 2023-01-11T22:05:26Z

There are only like two files that are letting their KeysInterface arguments be split up atomically, inbound_payment.rs and blinded_path.rs. Everything else percolates down to the Channel and ChannelManager struct, resulting in an all-or-nothing scenario.

TheBlueMatt · 2023-01-12T19:28:29Z

Can't OnionMessenger go separately as well? Also probably ChannelMonitor, no?

arik-so · 2023-01-12T19:41:33Z

Funny you say that, because those two are exactly the types I've been trying to make separate. But no, they both have dependency chains that suck in all of Channel and ChannelManager.

…ysInterface.

TheBlueMatt · 2023-01-13T19:23:41Z

If you remove the KeysInterface last, rather than first, I think it should work? Which dependency chain are you referring to?

arik-so · 2023-01-13T19:27:15Z

I'm talking about splitting up the keys_interface argument into implementations of its comprising traits. Once you have two or three arguments of different traits, that dependency chain sneaks through pretty much everything. That is unrelated to removing KeysInterface.

wpaulino · 2023-01-13T22:38:37Z

 	pub chain_monitor: &'b test_utils::TestChainMonitor<'c>,
 	pub keys_manager: &'b test_utils::TestKeysInterface,
-	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,
+	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,


Pre-existing, but we really need a type alias for this test ChannelManager...

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

arik-so · 2022-12-20T23:38:49Z

No description provided.

codecov-commenter · 2022-12-21T01:14:55Z

Codecov Report

Base: 90.73% // Head: 90.76% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (22d7a0b) compared to base (5221e4a).
Patch coverage: 97.60% of modified lines in pull request are covered.

❗ Current head 22d7a0b differs from pull request most recent head 49c1f30. Consider uploading reports for the commit 49c1f30 to get more accurate results

Additional details and impacted files

@@ Coverage Diff @@## main #1930 +/- ##
==========================================
+ Coverage 90.73% 90.76% +0.02% 
==========================================
Files 97 96 -1 Lines 50539 50088 -451 Branches 50539 50088 -451 ==========================================
- Hits 45859 45461 -398 + Misses 4680 4627 -53

Impacted Files	Coverage Δ
lightning-block-sync/src/init.rs	`91.11% <ø> (ø)`
lightning/src/chain/keysinterface.rs	`83.78% <ø> (-0.04%)`	⬇️
lightning/src/ln/chan_utils.rs	`93.56% <ø> (ø)`
lightning/src/ln/onion_route_tests.rs	`97.64% <ø> (-0.01%)`	⬇️
lightning/src/ln/payment_tests.rs	`98.73% <ø> (+<0.01%)`	⬆️
lightning/src/ln/peer_handler.rs	`55.82% <ø> (ø)`
lightning/src/ln/priv_short_conf_tests.rs	`96.54% <ø> (ø)`
lightning/src/ln/reorg_tests.rs	`100.00% <ø> (ø)`
lightning/src/ln/shutdown_tests.rs	`96.51% <ø> (ø)`
lightning/src/routing/router.rs	`90.90% <ø> (+0.03%)`	⬆️
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wpaulino · 2022-12-21T21:08:10Z

 	CW::Target: 'static + chain::Watch<<K::Target as SignerProvider>::Signer>,
 	T::Target: 'static + BroadcasterInterface,
-	K::Target: 'static + KeysInterface,
+	K::Target: 'static + EntropySource + NodeSigner + SignerProvider,


I imagine we can probably start to split each of these out into their own generic constraint (yay more generics). EntropySource seems like a good candidate since it's somewhat standalone. NodeSigner and SignerProvider really depends on the underlying deployment, since they will likely be backed by the same HSM/secure element.

Yea, if we're splitting the traits I have to assume we expect people to have different structs that implement different parts of the traits (otherwise why are we doing this to begin with :) ), in which case we need to break up the generics as well.

I agree we should split things up, but I would lean against doing all the splitting in this PR, maybe with the exception of some very shallow-level API examples, like BackgroundProcessor. BackgroundProcessor definitely lacks generics.

I don't think it will be less effort to do it in one PR? As-is is just more diff and intermediate state, it would be less total diff to review doing it all at once.

Well, one benefit of splitting it up into two is that it would allow @wpaulino to open his node_secret removal PR against the commit that's currently in this one, while I work on the follow-up in parallel. But I'll work on the parameter split tonight; maybe I can get it in fast enough and it's less trouble than I'm anticipating.

I don't think that would conflict, now that we split the traits and moved those functions around. This PR doesn't really heavily touch the keysinterface.rs file.

arik-so · 2022-12-22T21:23:33Z

Wilmer just made the point that instead of removing KeysInterface entirely, we could simply make it be comprised of NodeSigner and SignerProvider, removing its EntropySource component.

TheBlueMatt · 2022-12-22T22:05:42Z

I'm still not sure why its worth doing that as a separate PR, but, sure, if you want to do that to make forward progress, sounds fine. We'll ultimately need to do it for all of them anyway.

TheBlueMatt · 2023-01-03T17:00:34Z

This commit separation makes the PR really hard to review (ie generates more total lines of diff to look at) - can you instead break it up as individual commits which change different structs to use individual traits (including the lifetime changes) and then only as the very last commit remove the (at that point unused) KeysInterface trait? That should lead to a series of relatively small commits.

arik-so · 2023-01-05T20:34:53Z

I'll try

TheBlueMatt · 2023-01-05T22:13:15Z

Needs rebase after #1812, sorry. Should be mechanical but may be a bit annoying.

arik-so · 2023-01-11T22:05:26Z

There are only like two files that are letting their KeysInterface arguments be split up atomically, inbound_payment.rs and blinded_path.rs. Everything else percolates down to the Channel and ChannelManager struct, resulting in an all-or-nothing scenario.

TheBlueMatt · 2023-01-12T19:28:29Z

Can't OnionMessenger go separately as well? Also probably ChannelMonitor, no?

arik-so · 2023-01-12T19:41:33Z

Funny you say that, because those two are exactly the types I've been trying to make separate. But no, they both have dependency chains that suck in all of Channel and ChannelManager.

…ysInterface.

TheBlueMatt · 2023-01-13T19:23:41Z

If you remove the KeysInterface last, rather than first, I think it should work? Which dependency chain are you referring to?

arik-so · 2023-01-13T19:27:15Z

I'm talking about splitting up the keys_interface argument into implementations of its comprising traits. Once you have two or three arguments of different traits, that dependency chain sneaks through pretty much everything. That is unrelated to removing KeysInterface.

wpaulino · 2023-01-13T22:38:37Z

 	pub chain_monitor: &'b test_utils::TestChainMonitor<'c>,
 	pub keys_manager: &'b test_utils::TestKeysInterface,
-	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,
+	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,


Pre-existing, but we really need a type alias for this test ChannelManager...

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

arik-so · 2022-12-20T23:38:49Z

No description provided.

codecov-commenter · 2022-12-21T01:14:55Z

Codecov Report

Base: 90.73% // Head: 90.76% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (22d7a0b) compared to base (5221e4a).
Patch coverage: 97.60% of modified lines in pull request are covered.

❗ Current head 22d7a0b differs from pull request most recent head 49c1f30. Consider uploading reports for the commit 49c1f30 to get more accurate results

Additional details and impacted files

@@ Coverage Diff @@## main #1930 +/- ##
==========================================
+ Coverage 90.73% 90.76% +0.02% 
==========================================
Files 97 96 -1 Lines 50539 50088 -451 Branches 50539 50088 -451 ==========================================
- Hits 45859 45461 -398 + Misses 4680 4627 -53

Impacted Files	Coverage Δ
lightning-block-sync/src/init.rs	`91.11% <ø> (ø)`
lightning/src/chain/keysinterface.rs	`83.78% <ø> (-0.04%)`	⬇️
lightning/src/ln/chan_utils.rs	`93.56% <ø> (ø)`
lightning/src/ln/onion_route_tests.rs	`97.64% <ø> (-0.01%)`	⬇️
lightning/src/ln/payment_tests.rs	`98.73% <ø> (+<0.01%)`	⬆️
lightning/src/ln/peer_handler.rs	`55.82% <ø> (ø)`
lightning/src/ln/priv_short_conf_tests.rs	`96.54% <ø> (ø)`
lightning/src/ln/reorg_tests.rs	`100.00% <ø> (ø)`
lightning/src/ln/shutdown_tests.rs	`96.51% <ø> (ø)`
lightning/src/routing/router.rs	`90.90% <ø> (+0.03%)`	⬆️
... and 53 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

wpaulino · 2022-12-21T21:08:10Z

 	CW::Target: 'static + chain::Watch<<K::Target as SignerProvider>::Signer>,
 	T::Target: 'static + BroadcasterInterface,
-	K::Target: 'static + KeysInterface,
+	K::Target: 'static + EntropySource + NodeSigner + SignerProvider,


I imagine we can probably start to split each of these out into their own generic constraint (yay more generics). EntropySource seems like a good candidate since it's somewhat standalone. NodeSigner and SignerProvider really depends on the underlying deployment, since they will likely be backed by the same HSM/secure element.

Yea, if we're splitting the traits I have to assume we expect people to have different structs that implement different parts of the traits (otherwise why are we doing this to begin with :) ), in which case we need to break up the generics as well.

I agree we should split things up, but I would lean against doing all the splitting in this PR, maybe with the exception of some very shallow-level API examples, like BackgroundProcessor. BackgroundProcessor definitely lacks generics.

I don't think it will be less effort to do it in one PR? As-is is just more diff and intermediate state, it would be less total diff to review doing it all at once.

Well, one benefit of splitting it up into two is that it would allow @wpaulino to open his node_secret removal PR against the commit that's currently in this one, while I work on the follow-up in parallel. But I'll work on the parameter split tonight; maybe I can get it in fast enough and it's less trouble than I'm anticipating.

I don't think that would conflict, now that we split the traits and moved those functions around. This PR doesn't really heavily touch the keysinterface.rs file.

arik-so · 2022-12-22T21:23:33Z

Wilmer just made the point that instead of removing KeysInterface entirely, we could simply make it be comprised of NodeSigner and SignerProvider, removing its EntropySource component.

TheBlueMatt · 2022-12-22T22:05:42Z

I'm still not sure why its worth doing that as a separate PR, but, sure, if you want to do that to make forward progress, sounds fine. We'll ultimately need to do it for all of them anyway.

TheBlueMatt · 2023-01-03T17:00:34Z

This commit separation makes the PR really hard to review (ie generates more total lines of diff to look at) - can you instead break it up as individual commits which change different structs to use individual traits (including the lifetime changes) and then only as the very last commit remove the (at that point unused) KeysInterface trait? That should lead to a series of relatively small commits.

arik-so · 2023-01-05T20:34:53Z

I'll try

TheBlueMatt · 2023-01-05T22:13:15Z

Needs rebase after #1812, sorry. Should be mechanical but may be a bit annoying.

arik-so · 2023-01-11T22:05:26Z

There are only like two files that are letting their KeysInterface arguments be split up atomically, inbound_payment.rs and blinded_path.rs. Everything else percolates down to the Channel and ChannelManager struct, resulting in an all-or-nothing scenario.

TheBlueMatt · 2023-01-12T19:28:29Z

Can't OnionMessenger go separately as well? Also probably ChannelMonitor, no?

arik-so · 2023-01-12T19:41:33Z

Funny you say that, because those two are exactly the types I've been trying to make separate. But no, they both have dependency chains that suck in all of Channel and ChannelManager.

…ysInterface.

TheBlueMatt · 2023-01-13T19:23:41Z

If you remove the KeysInterface last, rather than first, I think it should work? Which dependency chain are you referring to?

arik-so · 2023-01-13T19:27:15Z

I'm talking about splitting up the keys_interface argument into implementations of its comprising traits. Once you have two or three arguments of different traits, that dependency chain sneaks through pretty much everything. That is unrelated to removing KeysInterface.

wpaulino · 2023-01-13T22:38:37Z

 	pub chain_monitor: &'b test_utils::TestChainMonitor<'c>,
 	pub keys_manager: &'b test_utils::TestKeysInterface,
-	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,
+	pub node: &'a ChannelManager<&'b TestChainMonitor<'c>, &'c test_utils::TestBroadcaster, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'b test_utils::TestKeysInterface, &'c test_utils::TestFeeEstimator, &'b test_utils::TestRouter<'c>, &'c test_utils::TestLogger>,


Pre-existing, but we really need a type alias for this test ChannelManager...

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

arik-so force-pushed the 2022-12-remove-keysinterface branch 2 times, most recently from c8fd161 to 74a5712Compare December 21, 2022 00:48

arik-so changed the title ~~WIP: Remove KeysInterface~~Remove KeysInterfaceDec 21, 2022

arik-so marked this pull request as ready for review December 21, 2022 03:37

arik-so requested a review from wpaulino December 21, 2022 03:37

tnull mentioned this pull request Dec 21, 2022
Add BDK-based KeysManager example lightningdevkit/lightningdevkit.org#183
Merged

arik-so requested a review from TheBlueMatt December 21, 2022 19:04

wpaulino reviewed Dec 21, 2022
View reviewed changes

arik-so force-pushed the 2022-12-remove-keysinterface branch 6 times, most recently from eafdaa2 to 22d7a0bCompare December 25, 2022 19:51

Remove KeysInterface trait.
5824e22

arik-so force-pushed the 2022-12-remove-keysinterface branch from 22d7a0b to a65fa82Compare January 12, 2023 23:58

arik-so added 2 commits January 12, 2023 16:10

Split up generic parameters that used to comprise KeysInterface.
72183bd

Decouple lifetimes for trait implementations that used to comprise Ke…
49c1f30
…ysInterface.

arik-so force-pushed the 2022-12-remove-keysinterface branch from a65fa82 to 49c1f30Compare January 13, 2023 00:11

wpaulino approved these changes Jan 13, 2023
View reviewed changes

wpaulino mentioned this pull request Jan 14, 2023
Remove get_node_secret from NodeSigner #1951
Merged

TheBlueMatt approved these changes Jan 14, 2023
View reviewed changes

TheBlueMatt merged commit ac6e0b3 into lightningdevkit:mainJan 14, 2023

TheBlueMatt mentioned this pull request Jan 14, 2023
Split up KeysInterface parameters into its comprising traits. #1952
Closed

carlaKC added a commit to carlaKC/rust-lightning that referenced this pull request Oct 8, 2024
arch: update diagram to reflect changes to KeysInterface
3c97480
KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

carlaKC added a commit to carlaKC/rust-lightning that referenced this pull request Oct 8, 2024
arch: update diagram to reflect changes to KeysInterface
4961ab5
KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.

carlaKC mentioned this pull request Oct 8, 2024
docs/trivial: update diagram to reflect changes to KeysInterface #3351
Merged

carlaKC added a commit to carlaKC/rust-lightning that referenced this pull request Oct 8, 2024
arch: update diagram to reflect changes to KeysInterface
7c1db56
KeysInterface was split into EntropySource, SignerProvider and NodeSigner in lightningdevkit#1930.