feat: Parquet Modular Encryption with Spark KMS for native readers by mbutrovich · Pull Request #2447 · apache/datafusion-comet

mbutrovich · 2025-09-23T19:34:34Z

Which issue does this PR close?

Closes #.

Rationale for this change

We want to add Parquet Module Encryption support for the native readers when using a Spark KMS. We use the encryption factory features added in DataFusion 50 to register an encryption factory that uses JNI to get decryption keys from Spark.

What changes are included in this PR?

How are these changes tested?

Existing PME tests with new readers added.
New tests that exercise PME options like plaintext footer, etc.

…ark side accessed via JNI.

codecov-commenter · 2025-09-23T19:58:41Z

Codecov Report

❌ Patch coverage is 36.78161% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.92%. Comparing base (f09f8af) to head (257f163).
⚠️ Report is 689 commits behind head on main.

Files with missing lines	Patch %	Lines
...rg/apache/comet/parquet/CometFileKeyUnwrapper.java	0.00%	18 Missing ⚠️
...a/org/apache/comet/parquet/CometParquetUtils.scala	0.00%	15 Missing ⚠️
...ain/scala/org/apache/comet/CometExecIterator.scala	33.33%	7 Missing and 1 partial ⚠️
...va/org/apache/comet/parquet/NativeBatchReader.java	0.00%	5 Missing ⚠️
...n/scala/org/apache/spark/sql/comet/operators.scala	80.76%	3 Missing and 2 partials ⚠️
...n/scala/org/apache/comet/rules/CometScanRule.scala	42.85%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #2447 +/- ##
============================================
+ Coverage 56.12% 58.92% +2.79% - Complexity 976 1457 +481 
============================================
Files 119 147 +28 Lines 11743 13642 +1899 Branches 2251 2369 +118 ============================================
+ Hits 6591 8038 +1447 - Misses 4012 4381 +369 - Partials 1140 1223 +83

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…yption factory registration in parquet_exec.rs.

parthchandra · 2025-09-24T22:07:15Z

Also look at https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/test/java/org/apache/parquet/crypto/TestPropertiesDrivenEncryption.java to see if there are any tests that might be relevant here.

# Conflicts: # spark/src/main/scala/org/apache/comet/CometExecIterator.scala

mbutrovich · 2025-09-30T17:02:27Z

Results attached from the benchmark I added to CometReadBenchmark, and a small chart with highlights to see what the overhead of encryption is for the various readers.

benchmark_decryption.txt

…and Factory caching.

parthchandra

lgtm

hsiang-c · 2025-10-06T17:46:23Z

+
+// spotless:off
+/*
+ * Architecture Overview:


This diagram is super helpful, thanks a lot.

# Conflicts: # native/core/src/execution/jni_api.rs # spark/src/main/scala/org/apache/comet/CometExecIterator.scala # spark/src/main/scala/org/apache/comet/Native.scala

hsiang-c

LGTM

…pache#2447)

Parquet Modular Encryption support for native readers using KMS on Sp…
dcc882b
…ark side accessed via JNI.

mbutrovich changed the title ~~feat: Parquet Modular Encryption support for native_datafusion and native_iceberg_compat readers~~feat: Parquet Modular Encryption with Spark KMS for native_datafusion and native_iceberg_compat readersSep 23, 2025

mbutrovich changed the title ~~feat: Parquet Modular Encryption with Spark KMS for native_datafusion and native_iceberg_compat readers~~feat: Parquet Modular Encryption with Spark KMS for native readersSep 23, 2025

Fix unused import.
8bac76a

hsiang-c reviewed Sep 23, 2025
View reviewed changes

Comment threadcommon/src/main/java/org/apache/comet/parquet/NativeBatchReader.java Outdated

hsiang-c reviewed Sep 23, 2025
View reviewed changes

Comment threadnative/core/src/parquet/parquet_exec.rs Outdated

hsiang-c reviewed Sep 23, 2025
View reviewed changes

Comment threadnative/core/src/parquet/parquet_exec.rs Outdated

mbutrovich added 4 commits September 23, 2025 18:06

Fix encryptionEnabled check in NativeBatchReader.java, and guard encr…
40935df
…yption factory registration in parquet_exec.rs.

Fix NPE when checking encryptedEnabled.
7cbfb1b

Merge branch 'main' into decryption
1e1fa2f

Minor refactor for encryptionEnabled.
090497b

mbutrovich added 7 commits September 26, 2025 10:38

Merge branch 'main' into decryption
992a4e1

More tests.
c9dfdd5

Cleanup Seq loop that wasn't doing anything.
bf0bec4

Merge branch 'main' into decryption
a0e2d9a

Docs.
271e940

Docs.
571c881

Refactor out of parquet_exec.rs.
4dde7fb

mbutrovich marked this pull request as ready for review September 26, 2025 20:31

mbutrovich added 2 commits September 29, 2025 10:13

Merge branch 'main' into decryption
ac566f5
# Conflicts: # spark/src/main/scala/org/apache/comet/CometExecIterator.scala

Add uniform encryption test.
9bc24fd

parthchandra reviewed Sep 29, 2025
View reviewed changes

mbutrovich added 3 commits September 30, 2025 07:47

Merge branch 'main' into decryption
1dfb252

Address PR feedback.
bf6ad03

Add benchmark.
7d1bf39

parthchandra reviewed Sep 30, 2025
View reviewed changes

Comment threadnative/core/src/parquet/encryption_support.rs
Comment threadcommon/src/main/java/org/apache/comet/parquet/CometFileKeyUnwrapper.java Outdated

martin-g reviewed Oct 1, 2025
View reviewed changes

Comment threadcommon/src/main/java/org/apache/comet/parquet/CometFileKeyUnwrapper.java Outdated
Comment threadnative/core/src/parquet/encryption_support.rs
Comment threadnative/core/src/parquet/encryption_support.rs Outdated

mbutrovich added 2 commits October 1, 2025 10:54

Address PR feedback related to number of hadoopConfs in a Comet plan …
8ba2680
…and Factory caching.

Adjust error handling.
e9fcca7

mbutrovich requested a review from parthchandra October 3, 2025 21:25

Add test with UNION.
e8c23ab

parthchandra approved these changes Oct 6, 2025
View reviewed changes

hsiang-c reviewed Oct 6, 2025
View reviewed changes

Comment threadspark/src/test/scala/org/apache/spark/sql/comet/ParquetEncryptionITCase.scala Outdated

hsiang-c reviewed Oct 6, 2025
View reviewed changes

Comment threadspark/src/test/scala/org/apache/spark/sql/comet/ParquetEncryptionITCase.scala

mbutrovich added 4 commits October 6, 2025 14:17

Add docs to reflect UNION discussion in PR feedback.
a53aa2f

Address PR feedback.
fbe2c96

Merge branch 'main' into decryption
d266184
# Conflicts: # native/core/src/execution/jni_api.rs # spark/src/main/scala/org/apache/comet/CometExecIterator.scala # spark/src/main/scala/org/apache/comet/Native.scala

Fix after merge conflicts with main.
257f163

hsiang-c approved these changes Oct 6, 2025
View reviewed changes

mbutrovich merged commit c23dc25 into apache:mainOct 7, 2025
102 checks passed

coderfender pushed a commit to coderfender/datafusion-comet that referenced this pull request Dec 13, 2025
feat: Parquet Modular Encryption with Spark KMS for native readers (a…
58ed4e4
…pache#2447)

mbutrovich mentioned this pull request Jan 16, 2026
Support client-side encryption apache/iceberg-rust#2034
Open

mbutrovich mentioned this pull request Feb 2, 2026
[iceberg] Add support to native_iceberg_compat for reading encrypted Parquet files #2202
Closed

mbutrovich deleted the decryption branch March 13, 2026 18:58

karuppayya mentioned this pull request May 14, 2026
feat: Credential provider support #4335
Closed

mbutrovich mentioned this pull request May 15, 2026
feat: vendor-pluggable S3 credentials for native scans #4309
Merged

mbutrovich mentioned this pull request Jul 21, 2026
feat: Iceberg table format V3: native table decryption, fall back for other V3 features #4991
Merged

Uh oh!

Conversation

mbutrovich commented Sep 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

codecov-commenter commented Sep 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

parthchandra commented Sep 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mbutrovich commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

parthchandra left a comment

Choose a reason for hiding this comment

Uh oh!

hsiang-cOct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hsiang-c left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mbutrovich commented Sep 23, 2025•
edited
Loading

codecov-commenter commented Sep 23, 2025•
edited
Loading