Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Make `max_total_cltv_expiry_delta` include the final CLTV by TheBlueMatt · Pull Request #1358 · lightningdevkit/rust-lightning · GitHub
Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Make `max_total_cltv_expiry_delta` include the final CLTV by TheBlueMatt · Pull Request #1358 · lightningdevkit/rust-lightning · GitHub
Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Make `max_total_cltv_expiry_delta` include the final CLTV by TheBlueMatt · Pull Request #1358 · lightningdevkit/rust-lightning · GitHub
Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Make `max_total_cltv_expiry_delta` include the final CLTV by TheBlueMatt · Pull Request #1358 · lightningdevkit/rust-lightning · GitHub
Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Make `max_total_cltv_expiry_delta` include the final CLTV by TheBlueMatt · Pull Request #1358 · lightningdevkit/rust-lightning · GitHub
Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Make `max_total_cltv_expiry_delta` include the final CLTV by TheBlueMatt · Pull Request #1358 · lightningdevkit/rust-lightning · GitHub
Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Make `max_total_cltv_expiry_delta` include the final CLTV by TheBlueMatt · Pull Request #1358 · lightningdevkit/rust-lightning · GitHub
Skip to content

Make max_total_cltv_expiry_delta include the final CLTV - #1358

Merged
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv
Mar 17, 2022
Merged

Make max_total_cltv_expiry_delta include the final CLTV#1358
TheBlueMatt merged 2 commits into
lightningdevkit:mainfrom
TheBlueMatt:2022-03-max-cltv

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

This fixes an integer underflow found by the router fuzz target
in CI.

@codecov-commenter

codecov-commenter commented Mar 10, 2022

Copy link
Copy Markdown

Codecov Report

Merging #1358 (b8c9029) into main (2bba1d4) will increase coverage by 1.25%.
The diff coverage is 62.50%.

Impacted file tree graph

@@ Coverage Diff @@## main #1358 +/- ##
==========================================
+ Coverage 90.60% 91.86% +1.25% 
==========================================
Files 72 73 +1 Lines 40324 47508 +7184 ==========================================
+ Hits 36536 43643 +7107 - Misses 3788 3865 +77 
Impacted FilesCoverage Δ
lightning/src/routing/router.rs94.03% <50.00%> (+1.63%)⬆️
lightning/src/ln/functional_tests.rs98.35% <100.00%> (+1.22%)⬆️
lightning/src/ln/priv_short_conf_tests.rs96.84% <0.00%> (ø)
lightning/src/ln/onion_route_tests.rs97.62% <0.00%> (+<0.01%)⬆️
lightning/src/ln/msgs.rs85.96% <0.00%> (+0.07%)⬆️
lightning/src/util/scid_utils.rs99.35% <0.00%> (+0.18%)⬆️
lightning/src/ln/functional_test_utils.rs96.66% <0.00%> (+1.33%)⬆️
lightning/src/ln/mod.rs96.96% <0.00%> (+1.96%)⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bba1d4...b8c9029. Read the comment docs.

@tnull

Copy link
Copy Markdown
Contributor

Yikes, of course the final CLTV delta needs to be accounted for at that point! My bad, glad the fuzzer caught this..
Thank you for the fix. LGTM (besides the failing benchmark)

@jkczyz
jkczyz self-requested a review March 10, 2022 21:12
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Added a new commit which fixes the benchmark.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5479 to +5481
for (first_hop, params, amt) in route_endpoints.iter() {
assert!(get_route(&payer, params, &graph.read_only(), Some(&[first_hop]), *amt, 42, &DummyLogger{}, &scorer, &random_seed_bytes).is_ok());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will significantly increase the benchmark runtime. I tried doing something similar when refactoring the benchmarks without much luck.

I have the benchmarks running on my machine now for over an hour and have only seen one result so far:

running 6 tests
test routing::router::benches::generate_mpp_routes_with_default_scorer ... bench: 10,138,522,397 ns/iter (+/- 541,770,074)

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grrr, I'd somehow thought the bencher was smarter than that :(.

Comment threadlightning/src/routing/router.rs Outdated
Comment on lines +5097 to +5105
let route = get_route(&our_id, &feasible_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes).unwrap();
let path = route.paths[0].iter().map(|hop| hop.short_channel_id).collect::<Vec<_>>();
assert_ne!(path.len(), 0);

// But not if we exclude all paths on the basis of their accumulated CLTV delta
let fail_max_total_cltv_delta = 23;
let fail_payment_params = PaymentParameters::from_node_id(nodes[6]).with_route_hints(last_hops(&nodes))
.with_max_total_cltv_expiry_delta(fail_max_total_cltv_delta);
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 42, Arc::clone(&logger), &scorer, &random_seed_bytes)
match get_route(&our_id, &fail_payment_params, &network_graph, None, 100, 0, Arc::clone(&logger), &scorer, &random_seed_bytes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make final_cltv_expiry_delta zero here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause the test was failing, and setting it to 0 is the most obvious change that doesn't change the test semantics at all by emulating the previous behavior.

Comment on lines -869 to +874
let max_total_cltv_expiry_delta = payment_params.max_total_cltv_expiry_delta
let max_total_cltv_expiry_delta = (payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta)
.checked_sub(2*MEDIAN_HOP_CLTV_EXPIRY_DELTA)
.unwrap_or(payment_params.max_total_cltv_expiry_delta);
.unwrap_or(payment_params.max_total_cltv_expiry_delta - final_cltv_expiry_delta);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help clarify my understanding, does $next_hops_cltv_delta include final_cltv_expiry_delta? Or is this needed because $candidate.cltv_expiry_delta() is actually for the previous hop as adjusted later on line 1534 and thus final_cltv_expiry_delta was not accounted for?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the latter is the case: at this point the final_cltv_expiry_delta has not been added and therefore needs to be accounted for. This is in contrast to later usages of payment_params.max_total_cltv_expiry_delta, as for example in add_random_cltv_offset(), where final_cltv_expiry_delta is already part of the paths' CTLV deltas.

@jkczyzjkczyz added this to the 0.0.106 milestone Mar 11, 2022
jkczyz
jkczyz previously approved these changes Mar 14, 2022
valentinewallace
valentinewallace previously approved these changes Mar 16, 2022
If the scoring in the routing benchmark causes us to take a
different path from the original scan, we may end up deciding that
the only path to a node has a too-high total CLTV delta, causing us
to panic in the benchmarking phase.
Here we simply check for that possibility and remove paths that
fail post-scoring.
This fixes an integer underflow found by the `router` fuzz target
in CI.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed without change.

@TheBlueMatt
TheBlueMatt merged commit c244c78 into lightningdevkit:mainMar 17, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@TheBlueMatt@codecov-commenter@tnull@jkczyz@valentinewallace