Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Add subcrate which impls a simple SPV client from Bitcoin Core RPC by TheBlueMatt · Pull Request #614 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add subcrate which impls a simple SPV client from Bitcoin Core RPC by TheBlueMatt · Pull Request #614 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add subcrate which impls a simple SPV client from Bitcoin Core RPC by TheBlueMatt · Pull Request #614 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Add subcrate which impls a simple SPV client from Bitcoin Core RPC by TheBlueMatt · Pull Request #614 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add subcrate which impls a simple SPV client from Bitcoin Core RPC by TheBlueMatt · Pull Request #614 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add subcrate which impls a simple SPV client from Bitcoin Core RPC by TheBlueMatt · Pull Request #614 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Add subcrate which impls a simple SPV client from Bitcoin Core RPC by TheBlueMatt · Pull Request #614 · lightningdevkit/rust-lightning · GitHub
Skip to content

Add subcrate which impls a simple SPV client from Bitcoin Core RPC - #614

Closed
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync
Closed

Add subcrate which impls a simple SPV client from Bitcoin Core RPC#614
TheBlueMatt wants to merge 2 commits into
lightningdevkit:masterfrom
TheBlueMatt:2020-05-rest-chainsync

Conversation

@TheBlueMatt

@TheBlueMattTheBlueMatt commented May 2, 2020

Copy link
Copy Markdown
Collaborator

This is still very much a WIP, and I want to reduce the dependencies to at least not rely on hyper as well as actually implement the REST-based client, though doing so should be trivial.

This adds a new subcrate lightning-http-blocks which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.

Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.

The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.

Addresses #627.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from aae0859 to 8fcb6e1CompareMay 2, 2020 04:42
These are essentially required to make rescan-at-reload doable as
individual ChannelMonitors may be synced to different chain states
and thus need to have blocks replayed separately.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 3 times, most recently from b0319cd to c872ef4CompareMay 2, 2020 05:27
@TheBlueMattTheBlueMatt changed the title WIP: Add subcrate which impls a simple SPV client from Bitcoin Core RPCAdd subcrate which impls a simple SPV client from Bitcoin Core RPCMay 2, 2020
@TheBlueMatt
TheBlueMatt marked this pull request as ready for review May 2, 2020 05:27
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Still a bit raw in terms of code quality and still need testing, but its a thing, and worth at least a glance-over.

@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 2 times, most recently from a5f78fc to a61c2cbCompareMay 2, 2020 19:01
@codecov

codecovBot commented May 2, 2020

Copy link
Copy Markdown

Codecov Report

Merging #614 into master will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #614 +/- ##
==========================================
- Coverage 91.12% 91.11% -0.01% 
==========================================
Files 34 34 Lines 20544 20544 ==========================================
- Hits 18720 18719 -1 - Misses 1824 1825 +1 
Impacted FilesCoverage Δ
lightning/src/ln/channelmonitor.rs95.50% <100.00%> (ø)
lightning/src/ln/functional_tests.rs97.02% <0.00%> (-0.02%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9098240...197e244. Read the comment docs.

@TheBlueMattTheBlueMatt mentioned this pull request May 3, 2020
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch 4 times, most recently from 0861e43 to 23368f5CompareMay 5, 2020 20:14
This adds a new subcrate `lightning-block-sync` which is designed
to make it easier to get up-and-running by removing the effort of
building an SPV client and fetching the chain.
Instead of building a P2P client (and all the address management
that entails), this focuses on building a trivial SPV client which
can fetch from several instances of an abstract BlockSource. Then,
we provide two example BlockSource implementations that can fetch
from Bitcoin Core's RPC interface and Bitcoin Core's REST interface.
The code here is taken with heavy modifications from
rust-lightning-bitcoinrpc.
@TheBlueMatt
TheBlueMattforce-pushed the 2020-05-rest-chainsync branch from 23368f5 to 197e244CompareMay 5, 2020 21:26
@valentinewallace
valentinewallace self-requested a review May 18, 2020 22:03

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't dived into the details yet but overall the structure looks good and makes sense!

I think the commits could be split up a bit more though. If you're busy with language bindings I don't mind taking on this task. The following split makes sense to me, in order:

  1. common utilities used between REST client and RPC client
  2. the HTTP client
  3. the REST client (or swap 2 and 3, or possibly combine them)
  4. BlockSource + its implementors + the structs that only BlockSource/its implementors use
  5. The AChainListener + its implementors
  6. The MicroSPVClient and its static functions

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also -- I think this could benefit from some example usage similar to what's in lightning-net-tokio: https://github.com/rust-bitcoin/rust-lightning/blob/master/lightning-net-tokio/src/lib.rs#L17 through line 62.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

@jkczyz

Copy link
Copy Markdown
Contributor

@valentinewallace if you feel up to it feel free to take this over! Or @jkczyz if so inclined.

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Regarding eec4beb, the refactoring mentioned above may alter this in some way. I'm looking into that now and will ping you with any questions that I may have.

@jkczyzjkczyz mentioned this pull request Jun 23, 2020
@jkczyz

jkczyz commented Jun 25, 2020

Copy link
Copy Markdown
Contributor

FYI, I've taken on this work and am currently working on some necessary refactoring to BlockNotifier and ChainListener.

Here's the PR rebased on #649 using BlockNotifier instead of AChainListener: https://github.com/jkczyz/rust-lightning/tree/pr-614-rebased.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Superseded by #763.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@jkczyz@valentinewallace@ariard