Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Support onion message pathfinding - #1669

Closed
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding
Closed

Support onion message pathfinding #1669
valentinewallace wants to merge 2 commits into
lightningdevkit:mainfrom
valentinewallace:2022-08-om-pathfinding

Conversation

@valentinewallace

Copy link
Copy Markdown
Contributor

We could reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

Partially addresses #1607

Used in upcoming commit(s) to test onion message pathfinding
We *could* reuse router::get_route, but it's better to start from scratch
because get_route includes a lot of payment-specific computations that impact
performance.

@TheBlueMattTheBlueMatt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Dijkstras implementation could be cleaned up a lot...

}

// Heavily adapted from https://github.com/samueltardieu/pathfinding/blob/master/src/directed/dijkstra.rs
// TODO: how2credit the repo (is that necessary?)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, good question. In general, the MIT and Apache licenses both require attribution, including of downstream projects. However, the Apache license only requires it if there is a file called "NOTICE" or any "copyright, patent, trademark, and attribution notices", which then must be provided downstream, and the MIT license only requires that "the above copyright notice be included", but the original repo doesn't actually include the MIT license anywhere, nor does it include any relevant notices as far as I can see, so there is no relevant "above copyright notice" to include, aside from the first paragraph of the MIT license, which we of course include as LICENSE-MIT. Thus, I think we can reasonably argue that a simple comment above this code indicating that it is adapted from (link) which is code Copyright Samuel Tardieu should suffice, which complies with the Apache "You must cause any modified files to carry prominent noticesstating that You changed the files" requirement.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1724 ended up not adapting this crate's implementation anymore (shoutout to Wikpedia)

Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
Comment threadlightning/src/onion_message/router.rs
@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

@valentinewallace
valentinewallace marked this pull request as draft August 17, 2022 17:49
@TheBlueMatt

Copy link
Copy Markdown
Collaborator

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

@tnull

tnull commented Aug 18, 2022

Copy link
Copy Markdown
Contributor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it.
If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

I believe the upstream dijkstras you pulled from is actually buggy, anyway, so the generic-ness isn't worth much :p

Lol I might've introduced bugs in my quest to eliminate all vecs from the original, to be fair

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Ok it's clearly less readable than I thought with a generic Dijkstra's. Staying generic would allow us to pretty much reuse the test vectors in the original repo, but it doesn't seem worth. Gonna convert to draft for now and rewrite the generic portions

Ah, didn't realize keeping it generic had that intention. Well, it's not unreadable and reusing test vectors may be worth it. If we however also see a way of refactoring the payment pathfinding to reuse the same generic implementation, we'd should definitely consider it, IMO.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol, we'd need a way to "decrement available liquidity" for hops in paths that were already found, at least, but maybe that's easy? 🤔

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

Oops, sorry lol didn't mean to close.

I did wonder if there were other cases where we'd want a generic implementation. I have no idea whether that'd be realistic with our payment pathfinding though lol

I kinda doubt it - we've ended up with a few algorithmic tweaks in addition to basic dijkstra's, so I'd be surprised if we could munge it back.

/// Find a path for sending an onion message.
pub fn find_path<L: Deref, GL: Deref>(
our_node_pubkey: &PublicKey, receiver_pubkey: &PublicKey, network_graph: &NetworkGraph<GL>, first_hops: Option<&[&PublicKey]>, logger: L
) -> Result<Vec<PublicKey>, LightningError> where L::Target: Logger, GL::Target: Logger

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we wouldn't reuse all the information we're learning and storing in our ProbabilisticScorer. I can see how we're limited with the current penalty being based on the link-level and here we might be interested by node-level reliability in the path construction. That said, we can also assume that a reliable channel == a reliable onion message communication channel and go with it. I don't know if the spec says anything here or what are the thinking of other implementations ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an option in the spec to return a separate error for "peer offline" from "no capacity", but I'm not sure how common that is. Once we land the historical scoring PR, we could use the "time in the zero-available-capacity bucket" as a score here.

@valentinewallace

Copy link
Copy Markdown
ContributorAuthor

Thanks for all the feedback! This needs a lot of work and I'm switching it up to work on custom OMs first -- closing for now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@valentinewallace@TheBlueMatt@tnull@ariard