aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

aead: stream module - #436

Merged
tarcieri merged 1 commit into
masterfrom
aead/stream
Jan 3, 2021
Merged

aead: stream module#436
tarcieri merged 1 commit into
masterfrom
aead/stream

Conversation

@tarcieri

Copy link
Copy Markdown
Member

Implementation of the STREAM construction as described in the paper "Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":

https://eprint.iacr.org/2015/189.pdf

The implementation is generic over AEAD ciphers and is factored into a low-level StreamPrimitive trait (permitting different "flavors" of STREAM) as well as higher-level stateful Encryptor and Decryptor objects which are generic over StreamPrimitive types.

Includes one concrete implementation of STREAM: StreamLE31, which uses a 31-bit counter and 1-bit last block flag. Note that this implementation differs slightly from the one described in the paper, which uses a 1-byte last block flag.

Using little endian provides better performance on commonly used CPU architectures, and using a 1-bit last block flag ensures the user-facing STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the construction described in the paper) and also avoids wasting bits.

It would probably make sense to provide a concrete implementation of STREAM as described in the paper as well, especially for compatibility with existing deployments of this construction, however I wanted to both make sure we provide a useful deployed STREAM variant as such, and also wanted to keep the PR smaller for initial review.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

Note: I left allocating APIs out of this PR as well to make it easier to review, but they're easily added in a follow-up.

@tarcieri
tarcieriforce-pushed the aead/stream branch 2 times, most recently from 874b73e to 1c1bfb9CompareDecember 26, 2020 21:38
Comment threadaead/src/stream.rs
Comment on lines +203 to +216
if self.position == S::COUNTER_MAX {
// Counter overflow
return Err(Error);
}

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this implementation precludes calling encrypt_next_in_place/decrypt_next_in_place with the maximum counter value.

That's deliberate: it ensures any segment encrypted under the maximum counter value MUST have the "last block" flag set.

Comment threadaead/src/stream.rs
#[doc = $obj_desc]
#[doc = "object in order to prevent further use."]
pub fn $last_method(
self,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consuming self here is annoying in async encryptors, where we may need to leave poll_close after encrypting the last chunk but before we've finished writing the encrypted chunk. It can be managed by storing an Option<aead::stream::Encryptor>, but it's a bit of a hassle.

More problematic is that consuming self here is incompatible with seeking decryptors, as we may need to decrypt the last chunk, read part of it, then seek back earlier than the last chunk. Decryptor doesn't implement Clone, so we can't clone before consuming. Instead, someone trying to implement Seek would need to save the key and nonce themselves, and then reconstruct the Decryptor every time a seek is requested.

@tarcieritarcieriDec 28, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Encryptor and Decryptor objects are high-level misuse resistant APIs designed to ensure the STREAM is encoded/decoded correctly.

The lower-level StreamPrimitive trait is intended for the use cases you're describing.

Comment threadaead/src/stream.rs
stream: S,

/// Current position in the STREAM.
position: S::Counter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no way to set position manually to a specific chunk, which completely prevents Decryptor from being used in a seeking context.

@str4dstr4d left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried migrating the age crate to this (which uses ChaCha20Poly1305 with a nonce structured as an 11-byte big-endian counter and 1-byte last block flag), and encountered a few issues.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d I think for something like age, you'll want to use StreamPrimitive directly. Encryptor and Decryptor are higher-level APIs intended to make it easy to do the right thing at the cost of flexibility.

They could potentially include seeking behavior similar to SyncStreamCipherSeek, but if you are intending to do anything in parallel the fact they keep state at all seems problematic and I think you should just use StreamPrimitive instead (which is deliberately immutable/stateless to handle such cases).

@str4d

Copy link
Copy Markdown

As I noted in Discord, StreamPrimitive doesn't actually offer me much for age at present, because I'm implementing both StreamPrimitive (to use age's nonce structure) and the things that would use StreamPrimitive (the most I could do was to replace my own enc/dec wrappers with Encryptor and Decryptor, but per above they don't completely meet my use case).

If we could move (what is currently) age's StreamReader and StreamWriter into a generic crate (abstracted so they can also e.g. support Tink's more general use-case), then it would make sense for age to use StreamPrimitive. I'm not sure whether it makes sense for those structs to belong in aead::stream, or whether they should be in another module or crate.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

@str4d it might make sense to spike things out in another crate initially, then circle back on upstreaming the parts that make sense into aead.

@tarcieritarcieri changed the title [WIP] aead: stream moduleaead: stream moduleJan 2, 2021
Implementation of the STREAM construction as described in the paper
"Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance":
https://eprint.iacr.org/2015/189.pdf
The implementation is generic over AEAD ciphers and is factored into
a low-level `StreamPrimitive` trait (permitting different "flavors"
of STREAM) as well as higher-level stateful `Encryptor` and `Decryptor`
objects which are generic over `StreamPrimitive` types.
Includes two concrete implementations of `StreamPrimitive`:
- `StreamBE32`: the original version of stream described in the paper,
with a nonce in the form: `prefix || counter || last_block`, where
`counter` is a 32-bit big endian-encoded integer, and `last_block`
is a 1-byte flag.
- `StreamLE31`: uses a 31-bit counter and 1-bit last block flag,
packed into the last 4 bytes of the nonce as a little endian integer.
Using little endian provides better performance on commonly used CPU
architectures, and using a 1-bit last block flag ensures the user-facing
STREAM nonce is even numbered in terms of bytes (e.g. for a 96-bit nonce
it'd be 64-bits or 8-bytes instead of a 7-byte nonce using the
construction described in the paper) and also avoids wasting bits.
@tarcieri
tarcieri marked this pull request as ready for review January 2, 2021 17:29
@tarcieri

Copy link
Copy Markdown
MemberAuthor

Added a StreamBE32 flavor of StreamPrimitive which implements the original construction as described in the paper, and marked this PR as ready for review

I'm wondering if we might reduce StreamPrimitive to just computing the nonce, rather than having an instance of the cipher, i.e. providing the aead_nonce method which is presently private.

That would reduce duplication of code between StreamBE32 and StreamLE31, and would allow StreamPrimitive to work with AeadMut, among other things.

@tarcieri

Copy link
Copy Markdown
MemberAuthor

I'm going to go ahead and land this as I think a stream module is both a valuable thing to have and something that comes up quite frequently.

I plan on submitting a follow-up PR to simplify the API as described above.

I'm also not in a rush to cut another release of aead and think we might consider making some breaking changes before the next release (e.g. #273), so I think at the very least it will have time to bake before that.

@tarcieri
tarcieri merged commit 92dc55f into masterJan 3, 2021
@tarcieri
tarcieri deleted the aead/stream branch January 3, 2021 16:12
This was referenced Feb 3, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@tarcieri@str4d