Fix race condition in async UtxoFuture resolution

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.

[tnull]

Fixes#4346.

Previously, we refactored the GossipVerifier to not require holding a
circular reference. As part of this, we moved to a model where the
UtxoFutures are now polled by the background processor which checks
for completion through get_and_clear_pending_msg_events.

However, as part of this refactor we introduced race-condition: as we
only held Weak references in PendingChecksContext and the
UtxoFuture was directly dropped by the GossipVerifier after calling
resolve, the actual data was dropped with the future and gone when the
background processor attempted to retrieve and apply it via
check_resolved_futures.

Here, we fix this issue by simply holding on to the stateArcs in a
separate pending_statesVec that is only pruned in
check_resolved_futures, ensuring any completed results are collected
first.

[ldk-reviews-bot]

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Now switched to a much less invasive approach where we simply keep track of the pending future states Arcs in a separate Vec until the next call of check_resolved_futures, which will collect completed states and prune the data structures.

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

[tnull]

Whoops, seems there are still some more tests to fix, drafting again 🤭

... and they're passing again.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (9e91b2e) to head (114f6b5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/utxo.rs	91.66%	23 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #4348 +/- ##
==========================================
+ Coverage 86.08% 86.10% +0.01% 
==========================================
Files 156 156 Lines 102428 102610 +182 Branches 102428 102610 +182 ==========================================
+ Hits 88179 88354 +175 - Misses 11754 11762 +8 + Partials 2495 2494 -1 

Flag	Coverage Δ
tests	86.10% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheBlueMatt]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[tnull]

Bleh, don't love the extra vec but you're right that its much simpler and probably worth it. Feel free to squash (maybe with a few more rustfmt'isms fixed) and let's land this.

Squashed fixup, and included the following changes:

diff --git a/lightning/src/routing/utxo.rs b/lightning/src/routing/utxo.rs
index f3351aa0a..ab653b1ea 100644
--- a/lightning/src/routing/utxo.rs+++ b/lightning/src/routing/utxo.rs@@ -763,22 +763,16 @@ mod tests {
network_graph.read_only().channels().get(&scid).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_none());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(!is_node_a_announced);
network_graph.update_node_from_announcement(&node_a_announce).unwrap();
network_graph.update_node_from_announcement(&node_b_announce).unwrap();
- assert!(network_graph- .read_only()- .nodes()- .get(&node_id_1)- .unwrap()- .announcement_info- .is_some());+ #[rustfmt::skip]+ let is_node_a_announced = network_graph.read_only().nodes().get(&node_id_1).unwrap()+ .announcement_info.is_some();+ assert!(is_node_a_announced);
}
@@ -1023,15 +1017,9 @@ mod tests {
assert!(!notifier_b.notify_pending());
network_graph.pending_checks.check_resolved_futures(&network_graph);
- assert!(!network_graph- .read_only()- .channels()- .get(&scid)- .unwrap()- .announcement_message- .as_ref()- .unwrap()- .contents- .features- .supports_unknown_test_feature());+ #[rustfmt::skip]+ let is_test_feature_set =+ network_graph.read_only().channels().get(&scid).unwrap().announcement_message+ .as_ref().unwrap().contents.features.supports_unknown_test_feature();+ assert!(!is_test_feature_set);
}

[TheBlueMatt]

thanks. just gonna land this it ended up being relatively trivial.

[TheBlueMatt]

Not really sure that rustfmt::skip is the right fix for these kinds of cases. We can take intermediate variables that clean up rustfmt's crap and improve readability while we're at it.