Backfill gossip without buffering directly in LDK by TheBlueMatt · Pull Request #1660 · lightningdevkit/rust-lightning

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt · 2022-08-10T00:13:49Z

This is intended as a pre-req for #1604 with a few of the changes from #1604 pulled in. The commit description of the "main" commit follows, with a few other minor tweaks present as well.

Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of PeerHandler and into the operating system buffer, where it
arguably belongs.

Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.

valentinewallace

Looks good, basically :)

valentinewallace · 2022-08-10T19:24:12Z

Looks good! I'm happy with this after we get a second reviewer. Feel free to squash

This consolidates our various checks on peer buffer space into the `Peer` impl itself, making the thresholds at which we stop taking various actions on a peer more readable as a whole. This commit was primarily authored by `Valentine Wallace <vwallace@protonmail.com>` with some amendments by `Matt Corallo <git@bluematt.me>`.

TheBlueMatt · 2022-08-10T19:29:52Z

Squashed without diff.

valentinewallace

ACK after squash

Instead of backfilling gossip by buffering (up to) ten messages at a time, only buffer one message at a time, as the peers' outbound socket buffer drains. This moves the outbound backfill messages out of `PeerHandler` and into the operating system buffer, where it arguably belongs. Not buffering causes us to walk the gossip B-Trees somewhat more often, but avoids allocating vecs for the responses. While its probably (without having benchmarked it) a net performance loss, it simplifies buffer tracking and leaves us with more room to play with the buffer sizing constants as we add onion message forwarding which is an important win. Note that because we change how often we check if we're out of messages to send before pinging, we slightly change how many messages are exchanged at once, impacting the `test_do_attempt_write_data` constants.

TheBlueMatt · 2022-08-15T21:35:12Z

Squashed without further changes.

valentinewallace · 2022-08-15T21:46:08Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

jkczyz · 2022-08-15T22:11:39Z

Ah, I think a warning may have been introduced in stable actually?

warning: variable does not need to be mutable
--> lightning/src/routing/gossip.rs:345:7
|
345 | let mut iter = if let Some(pubkey) = starting_point {
| ----^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default

Yeah, that's from addressing one of my comments. TIL you can avoid the explicit next call with this syntax:

diff --git a/lightning/src/routing/gossip.rs b/lightning/src/routing/gossip.rs
index cb49ab81..e91d251f 100644
--- a/lightning/src/routing/gossip.rs+++ b/lightning/src/routing/gossip.rs@@ -38,6 +38,7 @@ use io_extras::{copy, sink};
use prelude::*;
use alloc::collections::{BTreeMap, btree_map::Entry as BtreeEntry};
use core::{cmp, fmt};
+use core::ops::Bound;
use sync::{RwLock, RwLockReadGuard};
use core::sync::atomic::{AtomicUsize, Ordering};
use sync::Mutex;
@@ -342,14 +343,9 @@ where C::Target: chain::Access, L::Target: Logger
fn get_next_node_announcement(&self, starting_point: Option<&PublicKey>) -> Option<NodeAnnouncement> {
let nodes = self.network_graph.nodes.read().unwrap();
- let mut iter = if let Some(pubkey) = starting_point {- let mut iter = nodes.range(NodeId::from_pubkey(pubkey)..);- iter.next();- iter- } else {- nodes.range::<NodeId, _>(..)- };- for (_, ref node) in iter {+ let starting_bound = starting_point.map_or(+ Bound::Unbounded, |pubkey| Bound::Excluded(NodeId::from_pubkey(pubkey)));+ for (_, ref node) in nodes.range((starting_bound, Bound::Unbounded)) {
if let Some(node_info) = node.announcement_info.as_ref() {
if let Some(msg) = node_info.announcement_message.clone() {
return Some(msg);

TheBlueMatt · 2022-08-16T00:52:52Z

Squashed without further changes.

It's more accurate to name it as dropping gossip broadcasts, as it won't drop all gossip. Also fix accidental flipped bool introduced in lightningdevkit#1660

Fixes a flipped bool that was introduced in lightningdevkit#1660

TheBlueMatt mentioned this pull request Aug 10, 2022
Onion messages: add some initial rate limiting #1604
Merged

valentinewallace self-requested a review August 10, 2022 17:23

valentinewallace reviewed Aug 10, 2022
View reviewed changes

Comment threadlightning/src/ln/msgs.rs Outdated
Comment threadlightning/src/ln/peer_handler.rs Outdated
Comment threadlightning/src/ln/peer_handler.rs Outdated

TheBlueMatt force-pushed the 2022-08-cleanup-ratelimits branch from 4aaa673 to 92b767bCompare August 10, 2022 19:20

valentinewallace added the Seeking Code Review label Aug 10, 2022

valentinewallaceand others added 2 commits August 10, 2022 19:29

Clarify comment on BUFFER_DRAIN_MSGS_PER_TICK.
4f6da92

TheBlueMatt force-pushed the 2022-08-cleanup-ratelimits branch from 92b767b to 1bd0303Compare August 10, 2022 19:29

jkczyz self-requested a review August 11, 2022 18:09

tnull removed the Seeking Code Review label Aug 15, 2022

jkczyz reviewed Aug 15, 2022
View reviewed changes

Comment threadlightning/src/ln/peer_handler.rs Outdated
Comment threadlightning/src/ln/msgs.rs Outdated
Comment threadlightning/src/routing/gossip.rs Outdated
Comment threadlightning/src/routing/gossip.rs Outdated

TheBlueMatt assigned jkczyz and valentinewallace Aug 15, 2022

valentinewallace reviewed Aug 15, 2022
View reviewed changes

jkczyz previously approved these changes Aug 15, 2022
View reviewed changes

TheBlueMatt dismissed jkczyz’s stale review via 86dcb5bAugust 15, 2022 21:35

TheBlueMatt force-pushed the 2022-08-cleanup-ratelimits branch from 0d2504d to 86dcb5bCompare August 15, 2022 21:35

valentinewallace previously approved these changes Aug 15, 2022
View reviewed changes

valentinewallace requested a review from jkczyz August 15, 2022 21:38

jkczyz previously approved these changes Aug 15, 2022
View reviewed changes

TheBlueMatt dismissed stale reviews from jkczyz and valentinewallace via 7970f3dAugust 15, 2022 22:47

TheBlueMatt force-pushed the 2022-08-cleanup-ratelimits branch from 86dcb5b to 7970f3dCompare August 15, 2022 22:47

jkczyz previously approved these changes Aug 16, 2022
View reviewed changes

valentinewallace previously approved these changes Aug 16, 2022
View reviewed changes

TheBlueMatt added 2 commits August 16, 2022 00:52

Replace manual iteration with for loops in outbound gossip sync
0ee88d0

Drop uneccessary if {...; bool} pattern in PeerManager
8ffaacb

TheBlueMatt dismissed stale reviews from valentinewallace and jkczyz via 8ffaacbAugust 16, 2022 00:52

TheBlueMatt force-pushed the 2022-08-cleanup-ratelimits branch from 7970f3d to 8ffaacbCompare August 16, 2022 00:52

jkczyz approved these changes Aug 16, 2022
View reviewed changes

valentinewallace approved these changes Aug 16, 2022
View reviewed changes

TheBlueMatt merged commit 12687d7 into lightningdevkit:mainAug 16, 2022

valentinewallace mentioned this pull request Aug 24, 2022
PeerManager: add separate queue for gossip broadcasts #1683
Merged
1 task

valentinewallace added a commit to valentinewallace/rust-lightning that referenced this pull request Aug 25, 2022
PeerMan: fix bug in drop_gossip util
e14c7ed
Fixes a flipped bool that was introduced in lightningdevkit#1660