Add Parquet Modular encryption support (write)

[rok]

Which issue does this PR close?

Closes#7327

This PR is based on branch and an internal patch and /pull/6637 and aims to provide basic modular encryption support.

Rationale for this change

See #3511 and #7278. This allows users to write Parquet files with modular encryption.

What changes are included in this PR?

• Adds a new encryption::encrypt module, containing types for configuring encryption, and types that implement encryption of metadata and page data.
• Adds a new with_file_encryption_properties method to WriterPropertiesBuilder to enable encryption.
• Adds a new encryption test library, moves existing tests for reading encrypted Parquet files to this and adds new tests for writing encrypted files.

Are there any user-facing changes?

Yes, this adds the ability to specify encryption properties when writing Parquet files.

Several new types and methods are added when the encryption feature is enabled but there should be no breaking changes.

[adamreeve]

I came up with #7337 which I think is easier to work with

Thanks, this example helped. I took another pass at refactoring ThriftMetadataWriter and this is much nicer now.

Since this is a major release, we can make API changes too if needed to support modular encryption.

The one thing that comes to mind is whether we want to change AsyncArrowWriter::finish to return parquet::file::ParquetMetadata instead of parquet::format::FileMetaData. This was suggested in #7254 although closed without comment, but that would probably avoid the problem of writers not being able to use the returned metadata to retrieve statistics due to them being encrypted (https://github.com/apache/arrow-rs/pull/7111/files#r2009421835). But that might be quite a large and painful breaking change, and also might come with a performance hit...

[corwinjoy]

@alamb wrote:

Since this is a major release, we can make API changes too if needed to support modular encryption.

One minor API change that I would like to have considered is this related PR that I feel makes decryption usage better.
#7342

At the suggestion of @adamreeve I have posted this on its own PR.

[alamb]

But that might be quite a large and painful breaking change, and also might come with a performance hit...

We would have to try it I suspect to test

[alamb]

How is this PR doing, btw? Is it ready for another review?

[adamreeve]

Yes I think this is ready for review. There are a couple of comments I'd like some input on (https://github.com/apache/arrow-rs/pull/7111/files#r2015196618 and https://github.com/apache/arrow-rs/pull/7111/files#r2009421835) but otherwise it's looking good to me.

[alamb]

Thank you @rok@adamreeve and @corwinjoy 👏

I think this is looking really nice. I had some style comments but I don't think anything is required. I am not an expert in this area of the parquet spec, but I found the code clear, clean, and well commented and tested 🥇 🏆

I do also think it would be great to add some examples (as a follow on PR) that shows how to write an arrow record batch with encryption to a parquet file

The only thing I want to do before approving this is to run some benchmark tests (to make sure it doesn't accidentally slow down writing without encryption). I have started this and will report back

[alamb]

I ran the performance benchmarks and I do not see any changes for parquet writing with this PR (though the numbers do vary somewhat). Nice work. Details below

Details

++ critcmp main encryption-basics-fork
group encryption-basics-fork main
----- ---------------------- ----
write_batch nested/4096 values primitive list 1.00 6.5±0.30ms 325.6 MB/sec 1.03 6.7±0.26ms 316.1 MB/sec
write_batch nested/4096 values primitive list non-null 1.00 8.2±0.37ms 260.8 MB/sec 1.00 8.2±0.29ms 260.1 MB/sec
write_batch primitive/4096 values bool 1.01 1022.2±53.98µs 1062.3 KB/sec 1.00 1013.0±40.47µs 1072.0 KB/sec
write_batch primitive/4096 values bool non-null 1.00 934.1±27.14µs 627.3 KB/sec 1.03 960.6±34.00µs 610.0 KB/sec
write_batch primitive/4096 values float with NaNs 1.01 1824.6±56.19µs 30.1 MB/sec 1.00 1809.3±31.57µs 30.4 MB/sec
write_batch primitive/4096 values primitive 1.04 2.9±0.24ms 60.4 MB/sec 1.00 2.8±0.07ms 62.6 MB/sec
write_batch primitive/4096 values primitive non-null 1.06 2.9±0.25ms 60.5 MB/sec 1.00 2.7±0.13ms 64.0 MB/sec
write_batch primitive/4096 values primitive non-null with bloom filter 1.05 22.5±2.68ms 7.7 MB/sec 1.00 21.5±0.36ms 8.0 MB/sec
write_batch primitive/4096 values primitive with bloom filter 1.05 22.4±3.68ms 7.8 MB/sec 1.00 21.4±0.88ms 8.2 MB/sec
write_batch primitive/4096 values string 1.00 1746.6±55.52µs 72.3 MB/sec 1.01 1757.4±46.59µs 71.8 MB/sec
write_batch primitive/4096 values string dictionary 1.02 4.3±0.40ms 241.6 MB/sec 1.00 4.2±0.18ms 247.1 MB/sec
write_batch primitive/4096 values string dictionary with bloom filter 1.02 5.8±0.33ms 177.5 MB/sec 1.00 5.7±0.27ms 180.3 MB/sec
write_batch primitive/4096 values string non-null 1.02 6.9±0.24ms 295.6 MB/sec 1.00 6.8±0.25ms 302.8 MB/sec
write_batch primitive/4096 values string non-null with bloom filter 1.00 12.9±0.34ms 159.3 MB/sec 1.00 12.9±0.36ms 158.6 MB/sec
write_batch primitive/4096 values string with bloom filter 1.00 11.1±0.63ms 184.1 MB/sec 1.01 11.2±0.37ms 182.3 MB/sec

Second run

++ critcmp main encryption-basics-fork
group encryption-basics-fork main
----- ---------------------- ----
write_batch nested/4096 values primitive list 1.01 6.9±0.29ms 309.3 MB/sec 1.00 6.8±0.27ms 313.5 MB/sec
write_batch nested/4096 values primitive list non-null 1.00 8.5±0.41ms 251.6 MB/sec 1.04 8.8±0.51ms 241.4 MB/sec
write_batch primitive/4096 values bool 1.00 1012.3±34.35µs 1072.7 KB/sec 1.03 1044.5±39.21µs 1039.6 KB/sec
write_batch primitive/4096 values bool non-null 1.00 945.6±33.77µs 619.6 KB/sec 1.02 962.4±43.08µs 608.8 KB/sec
write_batch primitive/4096 values float with NaNs 1.01 1876.1±54.31µs 29.3 MB/sec 1.00 1861.8±63.39µs 29.5 MB/sec
write_batch primitive/4096 values primitive 1.03 3.0±0.13ms 58.5 MB/sec 1.00 2.9±0.12ms 60.4 MB/sec
write_batch primitive/4096 values primitive non-null 1.04 3.0±0.75ms 57.7 MB/sec 1.00 2.9±0.60ms 60.0 MB/sec
write_batch primitive/4096 values primitive non-null with bloom filter 1.00 21.5±0.50ms 8.0 MB/sec 1.00 21.5±0.65ms 8.0 MB/sec
write_batch primitive/4096 values primitive with bloom filter 1.00 21.4±0.85ms 8.2 MB/sec 1.00 21.4±0.96ms 8.2 MB/sec
write_batch primitive/4096 values string 1.00 1790.4±68.70µs 70.5 MB/sec 1.02 1827.0±87.24µs 69.1 MB/sec
write_batch primitive/4096 values string dictionary 1.00 4.6±0.28ms 225.3 MB/sec 1.06 4.8±0.30ms 213.4 MB/sec
write_batch primitive/4096 values string dictionary with bloom filter 1.00 6.2±0.31ms 166.9 MB/sec 1.01 6.2±0.42ms 165.3 MB/sec
write_batch primitive/4096 values string non-null 1.00 7.2±0.29ms 285.1 MB/sec 1.09 7.8±0.54ms 261.6 MB/sec
write_batch primitive/4096 values string non-null with bloom filter 1.01 13.4±0.41ms 152.7 MB/sec 1.00 13.3±0.42ms 153.9 MB/sec
write_batch primitive/4096 values string with bloom filter 1.00 11.9±0.48ms 172.3 MB/sec 1.00 11.9±0.35ms 172.7 MB/sec

[adamreeve]

Your latest comments should all be addressed now thanks @alamb. I've also just added one extra test to cover being able to use unencrypted row group metadata when writing a file.

I'll follow up with a PR soon to add some examples.

[alamb]

Thanks @adamreeve@corwinjoy and @rok ! I'll plan to merge this tomorrow

This PR appeared to have a conflict so I took the liberty of merging up from main

[alamb]

Merged to resolve another conflict 🙄

[alamb]

Thanks again @rok@etseidl@adamreeve and @corwinjoy 🦾 🚀

[alamb]

I also made a small PR to update the parquet status page (when this is released):

• Update Implementation Status to note that arrow-rs now fully supports modular encryption parquet-site#110

[alamb]

Follow on PR (for full support) is:

• Update Implementation Status to note that arrow-rs now fully supports modular encryption parquet-site#110 is released

I will mark that ready for review when we have released the next version of arrow-rs