Add Wycheproof tests and utilities

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.

[daviddrysdale]

If WYCHEPROOF_DIR environment variable is set, will attempt to find
a Wycheproof repo there.

If not, the build.rs script will clone the Wycheproof repo under
OUT_DIR.

[tarcieri]

Instead of putting these here, what if we published a wycheproof crate which could set up these sorts of tests in the individual repositories / CI flows for each of these crates?

[daviddrysdale]

Still work in progress, but I wanted to trigger the workflows and also ask:

• what's the best way to distribute the tests across repos/crates?
• how best to handle getting at a Wycheproof repo?

[tarcieri]

what's the best way to distribute the tests across repos/crates?

I'd suggest publishing a wycheproof crate containing any sort of shared code and adding myself as an owner.

The main way we do cross-crate tests today is via a dev feature of the respective trait crates in https://github.com/rustcrypto/traits, which mostly entails macros which write test cases. With that, the "tests" in the individual repos are just a macro invocation.

Here's an example:

https://github.com/RustCrypto/block-ciphers/blob/master/aes/tests/lib.rs

You could either pull the wycheproof crate in directly on the respective algorithm crates, or make the dev feature of the https://github.com/rustcrypto/traits crates pull in the wycheproof crate and have some algorithm family-specific setup for each of the test cases which are written by a macro ala the aes example above.

[newpavlov]

I think wycheproof should handle conversion of test vectors from the Wycheproof repository to our blb format. The resulting files then would be added into respective crates. It would mean that we would have to re-generate some blb files after the source repo gets updated, but also it would keep crate tests simple.

[tarcieri]

@newpavlov there's some information in the Wycheproof vector files which is absent from the .blb files which it'd be nice to retain, like a string describing the name of the vector and what it tests for, and whether or not the vector is expected to succeed or fail

[newpavlov]

@tarcieri
We have an ability to test for failure, e.g. in the AEAD tests we have status field. Also note that tests generated by dev macros report serial number of a failed test, so together with blb files we could generate TXT files containing description strings, so serial number of a failed tests would be equal to its description line number.

[tarcieri]

@newpavlov I guess if it could spit out a way to extract the failure using e.g. jq that’d work

[tarcieri]

@newpavlov what if wycheproof could run at a repo-level, spit out a good error message, but not bloat the actual crates?

The problem with the blobby tests is the UX is... not great:

Failed test №0

I'd really rather get the Wycheproof error message in these cases. They've done a really good job of documenting their test cases and what they represent and the failures actually mean.

[newpavlov]

We could add wycheproof-specific macros, which would contain an additional description field, but I don't think the UX improvement worth additional complexity and increased crate sizes (don't forget that crates are downloaded together with tests). Ideally we rarely will need these descriptions.

[tarcieri]

increased crate sizes (don't forget that crates are downloaded together with tests)

I was suggesting something more like how you've reorganized the criterion-based benchmarks, where we could have a publish = false pseudo-crate per repo containing the Wycheproof tests for every other crate in that repo, which gets run as part of CI.

It actually wouldn't be too far off from how this PR is currently structured, except by being in the same repo they'd get run for every PR.

[newpavlov]

I think it's worth to keep Wycheproof tests as part of tested crates. Not only will it keep CI setups simple, but also it will allow to catch possible regressions in third-party setups (e.g. during crater runs).

[tarcieri]

As long as there's some easy way to get the name of the failing test case, I'm okay with it

[daviddrysdale]

(Trying to get back to this, sorry for the delay.)

So would this be best reworked as a command-line tool that reads in Wycheproof data and emits .blb files?

Something like:

% w2blb --wycheproof_dir=/usr/src/wycheproof --algorithm=AES-SIV-CMAC --out=data/aes-siv.blb --txt=data/aes-siv.txt
% head -7 data/aes-siv.txt[0] = 'AES-SIV-CMAC case 296 [valid] empty message'[1] = 'AES-SIV-CMAC case 297 [valid] empty message'[2] = 'AES-SIV-CMAC case 298 [valid] empty message'[3] = 'AES-SIV-CMAC case 299 [valid] empty message'[4] = 'AES-SIV-CMAC case 300 [valid] empty message'[5] = 'AES-SIV-CMAC case 301 [valid] empty message'[6] = 'AES-SIV-CMAC case 302 [valid] message size divisible by block size'

If that is the preferred approach, are there any helpers for writing.blb files? (The blobby crate seems to just have read functions)

[newpavlov]

@daviddrysdale
Sounds good!

If that is the preferred approach, are there any helpers for writing .blb files?

Yes, see examples/convert.rs (we probably should move it to src/bin/). ./convert encode input.txt output.blb will convert hex-encoded files like this:

0123AAFF
FF

Each line (including empty-ones) will be converted to an entry in a generated blb file. For number of fields and their order you would have to look into dev modules of the respective trait crates. For example, for AEAD it's 6 fields in the following order: key, nonce, additional authenticated data, plaintext, ciphertext, and status (single byte, 0x00 for failure and 0x01 for success).

[daviddrysdale]

Started a separate PR over at #280

[daviddrysdale]

Now covered by #280 and various other PRs.