Surface bitcoind rpc error information by johncantrell97 · Pull Request #2057 · lightningdevkit/rust-lightning

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

johncantrell97 · 2023-02-28T16:41:23Z

Currently the RpcClient only surfaces bitcoind's rpc error message but not the error code. The code is useful downstream to more reliably understand what went wrong.

This creates a new struct RpcError that holds the code and message and then provides that as the inner payload to std::io::Error that is already being returned.

jkczyz · 2023-02-28T16:48:52Z

- let message = error["message"].as_str().unwrap_or("unknown error");
- return Err(std::io::Error::new(std::io::ErrorKind::Other, message));
+ let rpc_error = RpcError { 
+ code: error["code"].as_i64().unwrap_or(-1), 


Should we make RpcError::code an Option given -1 is RPC_MISC_ERROR? Possibly the same for message.

yeah I guess we could do that. I'm not sure what case (if ever) it won't have a code and message. I purposefully mapped it to RPC_MISC_ERROR since that's sort of what it is but happy to do it as Option's too if you think that makes more sense.

maybe it'd be a little cleaner to make RpcError an enum with a struct variant and an "Unknown" variant when there's no code/message 🤷‍♂️
I guess you don't gain much, still need to do a match to find out error code

Let's keep it as is. Like you said, it probably shouldn't ever happen and the code is used as a catch-all for when the RPC handling code throws an exception in Core.

Ok, Matt had made a comment that we should probably just use a new RpcError enum that wraps the io Error. think it’s worth just doing that?

That's a much larger change and is orthogonal, IMO, if you're just trying to include an extra piece of data. Changing the error type here means would make it inconsistent with RestCleint, and you'd ultimately need to wrap it again at the BlockSourceError level, anyhow, unless you want to change that as well. But that already is an enum.

Yeah, that makes sense. I was trying to avoid having to deal with a cascade of changes like that. If we're okay with this solution then it would work for what we needed (access to bitcoind error code)

jkczyz · 2023-02-28T18:07:11Z

Oh, could you update the commit message with context? We should make sure it's there independent of Github.

jkczyz · 2023-02-28T18:39:50Z


 let error = &response["error"];
 if !error.is_null() {
- // TODO: Examine error code for a more precise std::io::ErrorKind.


Looking into this a little more, I think the idea was that some errors may be considered transient while others are persistent. And this would affect the conversion done here:
rust-lightning/lightning-block-sync/src/convert.rs
Lines 21 to 29 in 8311581
implFrom<std::io::Error>forBlockSourceError{
fnfrom(e: std::io::Error) -> BlockSourceError{
match e.kind(){
std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
_ => BlockSourceError::transient(e),
}
}
}
This let's the user know if they should give-up or retry.

Ah ok, that makes sense. Well I can put the TODO back. Seems like to actually impl the TODO we would need to map the error rpc error codes into buckets for persistent vs transient.

Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

dunxen

LGTM with further specifying ErrorKind in a future PR.

jkczyz · 2023-03-02T14:26:16Z

Note that we currently check that the error contains some message strings in the sample client: https://github.com/lightningdevkit/ldk-sample/blob/e1a87a7e8533bb666ed8002032b124e628da8e12/src/bitcoind_client.rs#L261-L267

This should still work, but we may want to check the error code instead assuming that makes sense.

TheBlueMatt · 2023-03-02T19:19:14Z

I'm not really sure how we expect users to use this - in one specific case we happen to return a field inside the std::io::Error, so in theory they could downcast and do type-checking, but they have to know that its (maybe) there, which isn't documented anywhere afaict, and its not clear to me that std::io::ErrorKind::Other implies its there, so you can't just use that to check either.

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

jkczyz previously approved these changes Feb 28, 2023
View reviewed changes

jkczyz added Easy to Review Seeking Code Review labels Feb 28, 2023

jkczyz reviewed Feb 28, 2023
View reviewed changes

Surface bitcoind rpc error code
a7af24b
Users of the RpcClient had no way to access the error code returned by bitcoind's rpc. We embed a new RpcError struct as the inner error for the returned io::Error. Users can access both the code and the message using this inner struct.

johncantrell97 dismissed jkczyz’s stale review via a7af24bFebruary 28, 2023 19:06

johncantrell97 force-pushed the rpc-error-code branch from 5b39220 to a7af24bCompare February 28, 2023 19:06

jkczyz approved these changes Feb 28, 2023
View reviewed changes

dunxen approved these changes Mar 1, 2023
View reviewed changes

jkczyz removed the Seeking Code Review label Mar 2, 2023

jkczyz merged commit f114515 into lightningdevkit:mainMar 2, 2023

	implFrom<std::io::Error>forBlockSourceError{
	fnfrom(e: std::io::Error) -> BlockSourceError{
	match e.kind(){
	std::io::ErrorKind::InvalidData => BlockSourceError::persistent(e),
	std::io::ErrorKind::InvalidInput => BlockSourceError::persistent(e),
	_ => BlockSourceError::transient(e),
	}
	}
	}

Uh oh!

Conversation

johncantrell97 commented Feb 28, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkczyz commented Feb 28, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dunxen left a comment

Choose a reason for hiding this comment

Uh oh!

jkczyz commented Mar 2, 2023

Uh oh!

TheBlueMatt commented Mar 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

johncantrell97 commented Feb 28, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkczyz commented Feb 28, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dunxen left a comment

Choose a reason for hiding this comment

Uh oh!

jkczyz commented Mar 2, 2023

Uh oh!

TheBlueMatt commented Mar 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation