Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Hide InvoiceFeatures behind InvoiceBuilder API - #901

Merged
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics
May 4, 2021
Merged

Hide InvoiceFeatures behind InvoiceBuilder API#901
TheBlueMatt merged 6 commits into
lightningdevkit:mainfrom
jkczyz:2021-04-invoice-feature-semantics

Conversation

@jkczyz

@jkczyzjkczyz commented Apr 28, 2021

Copy link
Copy Markdown
Contributor

Instead of relying on users to set an invoice's features correctly, enforce the semantics inside InvoiceBuilder. For instance, if the user sets a PaymentSecret then InvoiceBuilder should ensure the appropriate feature bits are set. Thus, for this example, the TaggedField abstraction can be retained while still ensuring BOLT 11 semantics at the builder abstraction.

Based on #898.

signed_invoice: signed_invoice,
};
invoice.check_field_counts()?;
invoice.check_feature_bits()?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there cases where you may want to use this on an invoice not generated by us? Where we may want to accept a non-basic-mpp invoice but not generate them?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is supported as written. check_feature_bits simply checks that BOLT 11 requirements are met. Note that the changes in this PR don't require that any feature bits are set. I know we discussed doing that offline yesterday, but I ended up finding a simpler way to enforce the semantics.

}

impl<D: tb::Bool, H: tb::Bool, T: tb::Bool, C: tb::Bool> InvoiceBuilder<D, H, T, C, tb::False> {
/// Sets the payment secret and relevant features.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we still need to be able to set more features here, no? You may want to set some future features which is probably in known but not in the set that is required to understand the invoice here.

@jkczyzjkczyzApr 28, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional? If so, that is an open question that I had. For instance, would we want to have both basic_mpp_required and basic_mpp_optional methods? It wasn't entirely clear from the BOLT if either were possible:

  • if the basic_mpp feature is offered in the invoice:
    • MAY pay using Basic multi-part payments.
  • otherwise:
    • MUST NOT use Basic multi-part payments.

Does "offered" mean the odd bit (optional) was set? Is something similar possible for payment_secret? The BOLT seems to imply that if a secret is set then it must be used:

  • if there is a valid s field:
    • MUST use that as payment_secret

And what would optional mean here if basic_mpp is set as required as there is a dependency requirement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean setting a feature as optional?

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Does "offered" mean the odd bit (optional) was set?

I believe it means either. I think the handling of var_len_onion here is fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant a feature other than the two below. eg what if we have option_htlcs_colored_blue? Some users may really like blue HTLCs some users may not, and LDK may reasonably support receiving both. Thus, we need some way to provide a Features object, I think.

Ah, so the additional type parameter S is for specifying that a secret can be set at most once. Additionally, it is used to condition whether basic_mpp can bet set because that requires the payment_secret feature and thus also a secret field set in the invoice.

Other features may have similar type parameters associated with them if they require some other fields set in the invoice. For example, if the hypothetical feature were instead option_htlcs_colored with an additional field specifying the color blue, then we would add an option_htlcs_colored method taking a color to set in the invoice. It would also set the feature bits as necessary.

Long story short is the purpose of InvoiceBuilder is to enforce BOLT 11 semantics at compile time. Thus, it can never result in a SemanticsError only a CreationError. Allowing features to be set arbitrarily breaks that contract.

Whether the features are set as required or optional is an orthogonal concern. Since these features are optional in InvoiceFeatures::known(), then we'd likely want to set them optional here. I think I had decided on required because the dependency between them and wasn't sure what it would mean for some to be optional and others to be required.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, right, for "known" bits that makes sense. I do wonder how we can support "experimental" feature bits (eg DLC invoices or so), but we really need the ability to set them in InvoiceFeatures as well which would imply its a separate thing.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For experimental features, I'd imagine there could be a separate method to set these which could also check against any known features.

@codecov

codecovBot commented Apr 28, 2021

Copy link
Copy Markdown

Codecov Report

Merging #901 (718753d) into main (e26c3df) will increase coverage by 0.10%.
The diff coverage is 98.22%.

❗ Current head 718753d differs from pull request most recent head 2226ae2. Consider uploading reports for the commit 2226ae2 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## main #901 +/- ##
==========================================
+ Coverage 90.50% 90.60% +0.10% 
==========================================
Files 59 59 Lines 29624 30466 +842 ==========================================
+ Hits 26810 27604 +794 - Misses 2814 2862 +48 
Impacted FilesCoverage Δ
lightning-invoice/src/lib.rs90.32% <98.10%> (+2.73%)⬆️
lightning-invoice/src/utils.rs83.69% <100.00%> (ø)
lightning/src/ln/features.rs98.83% <100.00%> (+0.02%)⬆️
lightning/src/ln/functional_tests.rs97.03% <0.00%> (+0.22%)⬆️
lightning-invoice/src/ser.rs93.36% <0.00%> (+1.23%)⬆️
lightning-invoice/src/de.rs83.33% <0.00%> (+2.34%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e26c3df...2226ae2. Read the comment docs.

@devrandom

Copy link
Copy Markdown
Member

ACK modulo being able to set unknown feature bits

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

modulo being able to set unknown feature bits

I think we can do that in a followup? Currently InvoiceFeatures doesn't support custom bits at all, so there isn't a way to do it in invoice today anyway.

Feature payment_secret is required and depends on var_onion_optin, so
the latter must also be required.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch 2 times, most recently from 2ddb995 to 61c79deCompareApril 30, 2021 22:05
@jkczyz
jkczyz marked this pull request as ready for review April 30, 2021 22:06
@TheBlueMattTheBlueMatt added this to the 0.0.14 milestone May 1, 2021

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

Comment threadlightning-invoice/src/lib.rs
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

This is looking pretty reasonable to me! I'd prefer if test changes were included in the commit they're testing so it's a bit clearer what's being tested, but not a huge deal

I tend to agree though sometimes having a few atomic commits that culminate in the the tests can alleviate the review burden.

Here, the tests make use of InvoiceFeatures::known() which has the basic_mpp feature bit set. But the refactor involved removing that bit and then re-adding it in a later commit. I could have explicitly construct the features in the test, but I wanted the test to break if InvoiceFeatures::known() ever gets updated. That way InvoiceBuilder should always support our known features.

@ariardariard left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK 718753d

I don't think my point around basic_mpp matters given current spec.

self.tagged_fields = self.tagged_fields
.drain(..)
.map(|field| match field {
TaggedField::Features(f) => TaggedField::Features(f.set_basic_mpp_optional()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, what if in the future we introduce a new tagged field requiring to set the 9 field without being dependent on payment secret. We might set mpp here without actually having payment_secret set in our feature bits ?

Maybe we should add a if features.supports_payment_secret { ... } ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is conditionally implemented when payment_secret is set (i.e., when the S type parameter is tb::True). So supporting such a feature should not be problem -- calling basic_mpp() would be a compilation error still.

jkczyz added 4 commits May 3, 2021 16:23
Instead of relying on users to set an invoice's features correctly,
enforce the semantics inside InvoiceBuilder. For instance, if the user
sets a PaymentSecret then InvoiceBuilder should ensure the appropriate
feature bits are set. Thus, for this example, the TaggedField
abstraction can be retained while still ensuring BOLT 11 semantics at
the builder abstraction.
Since InvoiceFeatures are an implementation detail of InvoiceBuilder, an
explicit call is needed to support the basic_mpp feature. Since it is
dependent on the payment_secret feature, conditionally define the
builder's method only when payment_secret has been set.
@jkczyz
jkczyzforce-pushed the 2021-04-invoice-feature-semantics branch from 718753d to 2226ae2CompareMay 3, 2021 23:24
@jkczyz

Copy link
Copy Markdown
ContributorAuthor

Squashed fixup commits.

@ariard

Copy link
Copy Markdown

Code Review ACK 2226ae2

@TheBlueMatt
TheBlueMatt merged commit d782df0 into lightningdevkit:mainMay 4, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@jkczyz@devrandom@TheBlueMatt@ariard@valentinewallace