Replace `BTreeSet` in `IndexedMap` with sorted `Vec` by alecchendev · Pull Request #2040 · lightningdevkit/rust-lightning

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

alecchendev · 2023-02-18T20:06:36Z

Closes#1992.

Summarizing my findings:

I compared two ways of swapping BTreeSet for Vec:

Only sorting on calls to IndexedMap::range
Keeping it strictly sorted on inserts and removes

Both ways seemed to decrease memory usage of reading in a network graph from ~2.5MB to ~1.8MB, as well as very slightly improving the speed of read_network_graph. 2 had a significant regression for bench_reading_full_graph_from_file, so this PR is implementation 1: sorting on range.

alecchendev · 2023-02-18T20:08:23Z

Two things to note about this current implementation:

There are ~200 tests that have asserts comparing a network graph to an expected graph, so I reimplemented PartialEq on IndexedMap to not check for order now that IndexedMap::keys is no longer strictly sorted.
range now takes a &mut self which is kinda weird because logically it should be an immutable method. I hear people say cases like this might be a place to use interior mutability, but I didn't know if that would be overkill/mess up other things.

codecov-commenter · 2023-02-18T20:52:43Z

Codecov Report

Base: 87.22% // Head: 87.41% // Increases project coverage by +0.19% 🎉

Coverage data is based on head (62a88f9) compared to base (2f36c92).
Patch coverage: 88.08% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@ Coverage Diff @@## main #2040 +/- ##
==========================================
+ Coverage 87.22% 87.41% +0.19% 
==========================================
Files 100 101 +1 Lines 44051 45633 +1582 Branches 44051 45633 +1582 ==========================================
+ Hits 38424 39891 +1467 - Misses 5627 5742 +115

Impacted Files	Coverage Δ
lightning/src/ln/functional_test_utils.rs	`88.43% <0.00%> (-0.07%)`	⬇️
lightning/src/sync/fairrwlock.rs	`75.00% <0.00%> (ø)`
lightning/src/sync/mod.rs	`50.00% <50.00%> (ø)`
lightning/src/util/indexed_map.rs	`78.94% <72.22%> (+1.67%)`	⬆️
lightning/src/ln/outbound_payment.rs	`83.69% <77.77%> (+3.41%)`	⬆️
lightning/src/ln/payment_tests.rs	`95.86% <91.59%> (-0.08%)`	⬇️
lightning/src/sync/debug_sync.rs	`80.00% <92.85%> (-0.35%)`	⬇️
lightning/src/ln/channelmanager.rs	`87.66% <100.00%> (+1.81%)`	⬆️
lightning/src/routing/gossip.rs	`84.04% <100.00%> (+0.01%)`	⬆️
lightning/src/sync/nostd_sync.rs	`37.50% <100.00%> (+2.20%)`	⬆️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt

Nice, this is great!

TheBlueMatt

Looks good, sorry for the runaround on removing range.

The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

TheBlueMatt reviewed Feb 19, 2023
View reviewed changes

Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from eecff3d to 67cea68Compare February 19, 2023 02:43

TheBlueMatt reviewed Feb 20, 2023
View reviewed changes

Comment threadfuzz/src/indexedmap.rs Outdated
Comment threadlightning/src/util/indexed_map.rs

Replace BTreeSet in IndexedMap with sorted Vec
62a88f9
The `Vec` is sorted not on `IndexedMap::insert`, but on `IndexedMap::range` to avoid unnecessary work while reading a network graph.

alecchendev force-pushed the 2023-02-indexed-map-btreeset-to-vec branch from 67cea68 to 62a88f9Compare February 20, 2023 04:25

TheBlueMatt approved these changes Feb 20, 2023
View reviewed changes

TheBlueMatt added the Seeking Code Review label Feb 20, 2023

dunxen approved these changes Feb 21, 2023
View reviewed changes

TheBlueMatt merged commit 068c856 into lightningdevkit:mainFeb 21, 2023

Uh oh!

Conversation

alecchendev commented Feb 18, 2023

Uh oh!

alecchendev commented Feb 18, 2023

Uh oh!

codecov-commenter commented Feb 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alecchendev commented Feb 18, 2023

Uh oh!

alecchendev commented Feb 18, 2023

Uh oh!

codecov-commenter commented Feb 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alecchendev commented Feb 18, 2023

Uh oh!

alecchendev commented Feb 18, 2023

Uh oh!

codecov-commenter commented Feb 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

codecov-commenter commented Feb 18, 2023•
edited
Loading

codecov-commenter commented Feb 18, 2023•
edited
Loading

codecov-commenter commented Feb 18, 2023•
edited
Loading

codecov-commenter commented Feb 18, 2023•
edited
Loading

codecov-commenter commented Feb 18, 2023•
edited
Loading