Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Implement KeyObfuscator for Deterministic Encryption of storage keys. - #32

Merged
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate
Aug 16, 2024
Merged

Implement KeyObfuscator for Deterministic Encryption of storage keys.#32
G8XSU merged 2 commits into
lightningdevkit:mainfrom
G8XSU:obfuscate

Conversation

@G8XSU

@G8XSUG8XSU commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption, enhancing privacy and preventing common pitfalls (foot-guns) in client-side encryption.

Approach:

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together in vss-server.

Main reason for using ChaCha20-Poly1305:

  • Deterministic encryption
  • Already in dependency
  • We can generate synthetic iv ourselves.

Comparison with AES-256-SIV:

  • Deterministic encryption. (we need this for get operation to work, when client only knows about key.)
  • Designed for synthetic IV instead of random nonce. (we need this for get operation to work, when client only knows about key.)
  • Nonce misuse resistant, can even work without it.
  • Designed for key-wrapping problem in the first place. (we use this for nonce wrapping.)
  • Unlike just hashing, it is also reversible, this is needed for list operation to work, when client doesn't have any information about the keys.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. Obfuscating keys def. makes sense, but before I get into a detailed review, could you clarify why ChaCha20Poly1305 isn't an option here, in particular given that we already have working reviewed code for it and the explicity security warning given by the introduced aes-siv dependency? Or, if we go that route, why not go aes-gcm-siv which should be generally more secure and has a less scary security warning?

Also, it may be noteworthy that ChaCha20Poly1305 will likely soon get added to bitcoin_hashes (cf. rust-bitcoin/rust-bitcoin#2960) at which point we can drop our implementations here and in rust-lightning.

Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU

G8XSU commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

why ChaCha20Poly1305 isn't an option here

For chacha20, main problem was of nonce re-use and compromise of security guarantees when there is nonce re-use. Both chacha20 and aes-gcm were meant to work with random nonces. (aes-gcm-siv has more safety margin for nonce misuse but afaiu it is still vulnerable to nonce reuse and actively discourages it, whereas aes-siv was designed for key-wrapping problem and can even work without nonces. We are currently using key-wrapping to wrap nonces, it is secure as long as adversary doesn't know about the underlying plaintext and the key being wrapped isn't guessable.)

It is possible for the adversary to guess plaintext of certain keys(e.g. channel-manager).
I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

less scary security warning

Fwiw, both aes-gcm-siv and aes-siv have similar security warnings.
Underlying aes-gcm and aes implementation is reviewed/audited by NCC group. They do have the testvectors acc. to rfc, but aes-siv has additional warning regarding constant-time evaluation, which isn't as relevant for client-side encryption in this case but more so for secure enclave environment or similar.

@tnull

tnull commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

I was worried about security implications of using a deterministic method for encryption and nonce generation, with known nonce and guessable plaintext. (But on reconsideration, I think it should be fine if we use another derived-key as nonce for wrapping nonce. And nonce re-use risk is more applicable when we use it for different plaintexts iiuc.)

Right, I def. follow these concerns. But, if I'm not missing something, above sketched method of deterministically deriving the wrapped nonce's encryption nonce from the ciphertext and some key material should work to derive per-storage-key nonces allowing to use ChaCha20Poly1305.

In particular, we would have one nonce per plaintext (the storage key), i.e., if the adversary knows the plaintext they already know what we're trying to protect and will be able to determine the associated nonce but won't be able to learn more about nonces of other storage keys.

@G8XSU
G8XSU requested a review from tnullAugust 9, 2024 00:26

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for switching to ChaCha20Poly1305. Generally looks reasonable, a few comments.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
@G8XSU
G8XSU requested review from arik-so and tnullAugust 9, 2024 19:30
@tnull

Copy link
Copy Markdown
Contributor

This needs a rebase as we now have a duplicate dependency in Cargo.toml.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good to me, some comments/nits.

Would be great to see proptest support here to uncover and test edge cases.

Comment threadCargo.toml
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/key_obfuscator.rs
Comment threadsrc/util/key_obfuscator.rs Outdated
Comment threadsrc/util/mod.rs
Comment threadsrc/util/key_obfuscator.rs
Add KeyObfuscator to provide a helper object for client-side key obfuscation.
This implementation uses ChaCha20-Poly1305 for deterministic encryption and
tag authentication, enhancing security and preventing common pitfalls
(foot-guns) in client-side encryption.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side I think, should be ready for a second pair of eyes (cc @arik-so ?)

One question though: what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really? Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes indicating that it has been obfuscated and proceed without deobfuscation if the keys don't start with the correct sequence?

@G8XSU

Copy link
Copy Markdown
ContributorAuthor

what is the upgrade story here? Do we need to ship this and default to it before any users are going to use VSS really?

Yes the default is obfuscated keys for all users, this is a non-backward compatible change hence important to do before release.

Should we (otherwise on in general) prepend the ciphertext with some magic sentinel bytes

No, we cant do this because in 'get' operation we don't know whether key was obfuscated or not during 'put', hence all keys must be obfuscated.


// Wrap the synthetic nonce to store along-side key.
let (_, nonce_tag) = self.encrypt(&mut nonce, &ciphertext);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to recap (simplified) what happened here so far:

  1. We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
  2. We encrypt the plaintext P using the nonce N, getting the ciphertext C.
  3. We encrypt that nonce N using the ciphertext C as the nonce.

Is this approach cryptographically sound? Are there any risks associated with it?

@G8XSUG8XSUAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,
with minor correction, we use N2 = hash(ciphertext) to encrypt nonce N.

* We generate a nonce N from the plaintext storage key P, where basically N = hash(P).
* We encrypt the plaintext P using the nonce N, getting the ciphertext C.
* We encrypt that nonce N using the nonce N2 = hash (ciphertext-C) to obtain encrypted-nonce 'E'.
* We append ciphertext C and encrypted-nonce E , to store them together by sending in request to vss-server.

With using ChaCha20Poly1305 here, we need to take care of two things mainly:

  1. Don't re-use nonces for different plaintext. (Addressing Concern 1: We don't re-use it for different plaintexts, we do re-use it for same plaintext.)
  2. Since it is possible to guess some of the plaintexts like channel-manager, server shouldn't have access to enough information to be able to reverse the deterministic encryption.

For e.g. if we store unencrypted nonce N with cipher-text, and known plain text it might be possible to reverse it. Hence we store the encrypted nonce.

Addressing Concern 2: For nonce encryption we use hash(ciphertext) as nonce which isn't stored alongside data in server hence it should be safe.

@arik-so

Copy link
Copy Markdown

All right, thanks for the response! As far as I can tell, it looks good, too, and I am also slightly less concerned about the cryptography considering it's merely key obfuscation.

@G8XSU
G8XSU merged commit 52c1885 into lightningdevkit:mainAug 16, 2024
@G8XSUG8XSU mentioned this pull request Aug 23, 2024
G8XSU added a commit to G8XSU/vss-rust-client that referenced this pull request Aug 23, 2024
Major Changes include:
* Signature change in vss-client constructor. (in lightningdevkit#31 )
* Vss-client can now also return AuthError if AuthException is returned from server. (lightningdevkit#30)
* Adds VssHeaderProvider, can be used for auth and request signing.(lightningdevkit#31)
* Adds LnurlAuthToJwtProvider, provides LnUrl based JWT auth. (lightningdevkit#26)
* Adds KeyObfuscator, to provide client-side key obfuscation. (lightningdevkit#32)
* Package now has enforced MSRV of 1.63.0. (lightningdevkit#19)
This is a minor version bump because there are non-backward compatible changes in vss-client usage.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@G8XSU@tnull@arik-so