Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Pre-C bindings cleanups (2) - #638

Merged
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2
Jun 24, 2020
Merged

Pre-C bindings cleanups (2)#638
TheBlueMatt merged 10 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-06-c-bindings-cleanups-2

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

A second batch of assorted code cleanups that make C binding generation a bit easier. See individual commits for details, but they're all pretty straight forward.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch 3 times, most recently from a85467f to b006acbCompareJune 11, 2020 20:05
@codecov

codecovBot commented Jun 11, 2020

Copy link
Copy Markdown

Codecov Report

Merging #638 into master will decrease coverage by 0.07%.
The diff coverage is 89.54%.

Impacted file tree graph

@@ Coverage Diff @@## master #638 +/- ##
==========================================
- Coverage 91.30% 91.23% -0.08% 
==========================================
Files 35 35 Lines 21104 21114 +10 ==========================================
- Hits 19270 19263 -7 - Misses 1834 1851 +17 
Impacted FilesCoverage Δ
lightning/src/ln/msgs.rs90.03% <ø> (ø)
lightning/src/util/ser.rs87.45% <ø> (ø)
lightning/src/util/test_utils.rs85.20% <0.00%> (ø)
lightning/src/routing/network_graph.rs91.06% <38.09%> (-0.56%)⬇️
lightning/src/ln/channel.rs86.76% <75.00%> (-0.03%)⬇️
lightning/src/chain/chaininterface.rs91.93% <100.00%> (+0.04%)⬆️
lightning/src/chain/keysinterface.rs94.78% <100.00%> (ø)
lightning/src/ln/chanmon_update_fail_tests.rs97.45% <100.00%> (ø)
lightning/src/ln/channelmanager.rs85.45% <100.00%> (ø)
lightning/src/ln/channelmonitor.rs95.70% <100.00%> (+<0.01%)⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fca07...5c37023. Read the comment docs.

@jkczyz
jkczyz self-requested a review June 12, 2020 18:16

@jkczyzjkczyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Comment on lines +238 to +239
{
let funding_txo = monitor.get_funding_txo();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this scope necessary because of the borrow here?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is for 1.22, but not for newer versions.

for (index, transaction) in block.txdata.iter().enumerate() {
if self.does_match_tx_unguarded(transaction, &watched) {
matched.push(transaction);
matched_index.push(index as u32);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep this as usize and return Vec<usize> since you are just casting it back at the call site.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did so in a new commit since it had a bunch of followon effects.

Comment threadlightning/src/ln/functional_tests.rs Outdated
let (_, our_payment_hash) = get_payment_preimage_hash!(nodes[0]);
let net_graph_msg_handler = &nodes[1].net_graph_msg_handler;
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), net_graph_msg_handler, &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();
nodes[1].node.send_payment(&get_route(&nodes[1].node.get_our_node_id(), &*net_graph_msg_handler.network_graph.read().unwrap(), &nodes[0].node.get_our_node_id(), None, &Vec::new(), 40000, TEST_FINAL_CLTV, &logger).unwrap(), our_payment_hash, &None).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dereference is unnecessary here and throughout.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The types are different - without the ref+deref its a MutexGuard, with the ref+deref you get the NetworkGraph itself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to explicitly dereference a MutexGuard. See how it is working correctly in fuzz/src/full_stack.rs without the dereference.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not sure if that is the same case, but it does compile here if you remove the dereference.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'd tried previously without either the & or *, which obviously doesn't work, but you're right, works with only the &.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 912d57a to ef57682CompareJune 13, 2020 18:12
@jkczyz

Copy link
Copy Markdown
Contributor

The changes related to get_funding_txo in 0acdc55 belong in 897a1ab.

Not sure if you saw my previous top-level comment (quoted here) but otherwise looks good.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from ef57682 to 020ce15CompareJune 13, 2020 21:34
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

I did not. Good catch, fixed.

@jkczyz

Copy link
Copy Markdown
Contributor

Looks like a few places in the fuzz code need to be updated for 020ce15.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 795ae9c to 7741f38CompareJune 16, 2020 19:56

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial review

Comment threadlightning/src/routing/network_graph.rs
Comment threadlightning/src/routing/network_graph.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just some nits

Comment threadlightning/src/ln/channel.rs Outdated
channel_id: self.channel_id(),
data: "funding tx had wrong script/value".to_owned()
});
} else if txo_idx > 0xff_ff {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these checks be tested? or a TODO, or issue to test them?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, duh, funding_txo.index is a u16, so this line is unreachable.

... instead of only the txid.
This is another instance of it not being possible to fully
re-implement SimpleManyChannelMonitor using only public methods. In
this case you couldn't properly register outpoints for monitoring
so that the funding transaction would be matched.
In general, we don't need an explicit lifetime when doing something
like:
fn get_thing(&self) -> &Thing { &self.thing }.
This also makes it easier to reason about what's going on in the
bindings generation.
This isn't a big difference in the API, but it avoids needing to
wrap a given NetworkGraph in a RwLock before passing it, which
makes it much easier to generate C bindings for.
This is more consistent with the way we use std::cmp over the
codebase and avoids `use std`, which is only actually needed to
support older rustcs, so feels a bit awkward.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 7741f38 to 8b4c5a3CompareJune 23, 2020 04:09
non-mut references to primitives are only excess overhead, so
there's not much reason to ever have them. As a nice bonus, it also
is one less thing to worry about when generating C bindings
Instead of making the filter_block fn in the ChainWatchInterface
trait return both a list of indexes of transaction positions within
the block and references to the transactions themselves, return
only the list of indexes and then build the reference list at the
callsite.
While this may be slightly less effecient from a memory locality
perspective, it shouldn't be materially different.
This should make it more practical to generate bindings for
filter_block as it no longer needs to reference Rust Transaction
objects that are contained in a Rust Block object (which we'd
otherwise just pass over the FFI in fully-serialized form).
This was just an oversight when route calculation was split up into
parts - it makes no sense for get_route to require that we have a
full route message handler, only a network graph (which can always
be accessed from a NetGraphMsgHandler anyway).
We use them largely as indexes into a Vec<Transaction> so there's
little reason for them to be u32s. Instead, use them as usize
everywhere.
We also take this opportunity to add range checks before
short_channel_id calculation, as we could otherwise end up with a
bogus short_channel_id due to an output index out of range.
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-06-c-bindings-cleanups-2 branch from 8b4c5a3 to 5c37023CompareJune 23, 2020 20:13

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, updates LGTM

@TheBlueMatt
TheBlueMatt merged commit 8fae0c0 into lightningdevkit:masterJun 24, 2020
@jkczyzjkczyz mentioned this pull request Jun 25, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt@jkczyz@valentinewallace