Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Use interior mutability in ChannelMonitor by jkczyz · Pull Request #813 · lightningdevkit/rust-lightning · GitHub
Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Use interior mutability in ChannelMonitor by jkczyz · Pull Request #813 · lightningdevkit/rust-lightning · GitHub
Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Use interior mutability in ChannelMonitor by jkczyz · Pull Request #813 · lightningdevkit/rust-lightning · GitHub
Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Use interior mutability in ChannelMonitor by jkczyz · Pull Request #813 · lightningdevkit/rust-lightning · GitHub
Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Use interior mutability in ChannelMonitor by jkczyz · Pull Request #813 · lightningdevkit/rust-lightning · GitHub
Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Use interior mutability in ChannelMonitor by jkczyz · Pull Request #813 · lightningdevkit/rust-lightning · GitHub
Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Use interior mutability in ChannelMonitor by jkczyz · Pull Request #813 · lightningdevkit/rust-lightning · GitHub
Skip to content

Use interior mutability in ChannelMonitor - #813

Merged
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex
Mar 2, 2021
Merged

Use interior mutability in ChannelMonitor#813
TheBlueMatt merged 4 commits into
lightningdevkit:mainfrom
jkczyz:2021-02-channel-monitor-mutex

Conversation

@jkczyz

@jkczyzjkczyz commented Mar 1, 2021

Copy link
Copy Markdown
Contributor

ChainMonitor holds a map of ChannelMonitors guarded by a single Mutex. This prevents parallelizing update_channel operations. This PR moves the Mutex inside ChannelMonitor such that it uses interior mutability, making its interface in terms of &self rather than &mut self. Thus, ChainMonitor can use a RwLock for its map instead of a Mutex, allowing for parallel operations other than when updating the map in watch_channel.

Additionally, since ChannelMonitor now uses interior mutability, it can implement chain::Listen without using a RefCell.

Resolves#805

@jkczyz
jkczyz requested a review from TheBlueMattMarch 1, 2021 07:08
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

@TheBlueMatt Not sure if this is what you had in mind with moving Mutex inside ChannelMonitor. It was definitely the simplest way of accomplishing this, but it means multiple operations on a ChannelMonitor require obtaining the lock more than once. Additionally, get_funding_txo and get_outputs_to_watch need to clone their output since a reference from the locked struct cannot be returned.

@codecov

codecovBot commented Mar 1, 2021

Copy link
Copy Markdown

Codecov Report

Merging #813 (e8ea0d9) into main (9fba7c9) will decrease coverage by 0.05%.
The diff coverage is 96.00%.

Impacted file tree graph

@@ Coverage Diff @@## main #813 +/- ##
==========================================
- Coverage 91.06% 91.01% -0.06% 
==========================================
Files 48 48 Lines 25495 25562 +67 ==========================================
+ Hits 23218 23266 +48 - Misses 2277 2296 +19 
Impacted FilesCoverage Δ
lightning-block-sync/src/init.rs93.56% <ø> (ø)
lightning/src/util/macro_logger.rs88.33% <ø> (ø)
lightning/src/chain/channelmonitor.rs95.52% <94.61%> (-0.24%)⬇️
lightning/src/chain/chainmonitor.rs94.20% <100.00%> (+0.08%)⬆️
lightning/src/ln/chanmon_update_fail_tests.rs97.56% <100.00%> (ø)
lightning/src/ln/channel.rs87.83% <100.00%> (ø)
lightning/src/ln/functional_test_utils.rs94.97% <100.00%> (ø)
lightning/src/ln/functional_tests.rs96.92% <100.00%> (-0.23%)⬇️
lightning/src/util/test_utils.rs83.49% <100.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fba7c9...e8ea0d9. Read the comment docs.

@TheBlueMattTheBlueMatt added this to the 0.0.13 milestone Mar 1, 2021
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Concept ACK.

@valentinewallace

Copy link
Copy Markdown
Contributor

Concept ACK :)

@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 2d42d4d to 7aa17e6CompareMarch 1, 2021 20:43
@jkczyz
jkczyz marked this pull request as ready for review March 1, 2021 20:43
///
/// panics if the given update is not the next update by update_id.
pub fn update_monitor<B: Deref, F: Deref, L: Deref>(
&self,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I find rustfmt-style unwrapping like this rather hard to read, its something that my head thinks shouldn't be vertical but is (and I have to check it against the use which is horizontal), and suddenly I can't see as much context.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was resistant to this for awhile until dealing with extremely long signatures in lightning-block-sync. I think using this format for long signatures is a good convention because it makes the parameters easy to scan and provides a consistent rule for formatting.

Otherwise, parameters can be easily missed if they are bin packed. It may seem a little odd at first, but it doesn't take long to adjust to reading this format. And it has the advantage of clearly indicating a parameter's type. This otherwise becomes especially difficult for more elaborate types like tuples or those with more than one type parameter, as the additional commas make the parameter list harder to scan.

That said, I do prefer to bin pack arguments at the call sites as I've done throughout in cases where the line exceeds the 100-character limit. Here, there are no types so any advantage of breaking each argument across lines is lost. I think this is a good middle ground as the additional vertical spacing is limited to signatures.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is somewhere in between, indeed, and I see the value of it, but I'd argue that more things are missed in review due to missing context than scan errors. Sometimes just a few lines is needed, but sometimes even the next function up is important (or being able to fit the full parameter list while looking at a function). Sadly, different people have different terminal heights so there's not an easy fix.

}
}

#[cfg(test)]

@valentinewallacevalentinewallaceMar 1, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, why are these functions test-only now?

Edit: oh, I guess that does make sense. In that case, maybe we could make the inner's corresponding methods private, instead of pub(crate)?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point. Done!

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the public API so much nicer

jkczyz added 4 commits March 1, 2021 22:12
ChainMonitor accesses a set of ChannelMonitors behind a single Mutex.
As a result, update_channel operations cannot be parallelized. It also
requires using a RefCell around a ChannelMonitor when implementing
chain::Listen.
Moving the Mutex into ChannelMonitor avoids these problems and aligns it
better with other interfaces. Note, however, that get_funding_txo and
get_outputs_to_watch now clone the underlying data rather than returning
references.
Now that ChannelMonitor uses an internal Mutex to support interior
mutability, ChainMonitor can use a RwLock to manage its ChannelMonitor
map. This allows parallelization of update_channel operations since an
exclusive lock only needs to be held when adding to the map in
watch_channel.
The implementation of chain::Listen for ChannelMonitor required using a
RefCell since its block_connected method required a mutable borrow. This
is no longer the case since ChannelMonitor now uses interior mutability
via a Mutex. So the RefCell is no longer needed.
@jkczyz
jkczyzforce-pushed the 2021-02-channel-monitor-mutex branch from 7aa17e6 to e8ea0d9CompareMarch 2, 2021 06:16
@TheBlueMatt
TheBlueMatt merged commit e241ca4 into lightningdevkit:mainMar 2, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add inner mutexes to ChannelMonitor

3 participants

@jkczyz@TheBlueMatt@valentinewallace