API, Core: Introduce a generic abstraction to replace ManifestListFile by gaborkaszab · Pull Request #17545 · apache/iceberg

gaborkaszab · 2026-08-07T09:20:30Z

ManifestListFile and its implementation contains nothing that is specific to manifest lists. It is more generally related to files that use TableMetadata.encryptionKeys to store encrypted encryption key metadata that are referred to by a key ID.

This PR introduces a more general interface that can be used accross multiple file types like manifest lists, V4 root manifests, table statistics and partition statistics. The less general functionality specific to manifest lists is deprecated or removed where possible.

gaborkaszab · 2026-08-07T10:15:47Z

+ * @deprecated will be removed in 2.0.0; use {@link #newInputFile(EncryptableFile)} instead.
+ */
+ @Deprecated
+ default InputFile newInputFile(ManifestListFile manifestList) {


I dropped BaseManifestListFile because that was package private. Now, there is no implementation in the library that can call this function, still I don't think we can drop this, because it'd break API for users that happen to implement their own ManifestListFile. Not likely, but technically feasible.

stevenzwu · 2026-08-07T15:43:40Z

+ * A file that may be encrypted. If it is encrypted, its encrypted key metadata is tracked in the
+ * table metadata encryption keys and is referenced by a key ID.
+ */
+public interface EncryptableFile {


Naming: "Encryptable" describes a capability every file has, not the specific concept/state represented here. I would suggest EncryptedFile and let the Javadoc cover the "may or may not carry a keyId" case.

Changed to EncryptedFile, however, I'm still not comfortable with the vile name. The name suggests the file is encrypted, but it's may or may not. Maybe FileWithEncryptedKey ?

let's see if others have a better name than EncryptedFile.
We do have PlaintextEncryptionManager for no encryption too.

I feel FileWithEncryptedKey is more accurate.

Renamed the interface to FileWithEncryptedKey

stevenzwu · 2026-08-07T15:43:49Z

+ * The file key metadata can be encrypted. Returns ID of encryption key or null if it's not
+ * encrypted.
+ */
+ String encryptionKeyID();


encryptionKeyID() → keyId() for consistency with Snapshot.keyId() and EncryptedKey.keyId() (ID caps is out of style — even the local variables in EncryptionUtil are manifestListKeyId).
Javadoc suggestion: /** Returns the encryption key ID for this file, or null if the file is not encrypted. */ — the interface-level javadoc already covers the mapping to TableMetadata.encryptionKeys(), so the method sentence can stay short.

Thanks for the comment suggestion! Applied.
About the function name: I'm not sure we can change the function name, because that would break the API for ManifestListFile that is derived from this interface.
Alternatively, we can avoid deriving ManifestListFile from the new interface, but then we won't be able to cast it to the new interface and delegate to the new function signature like in FileIO or in EncryptionUtil:
newInputFile((EncryptableFile) manifestList)

if we want to, we can still rename it to a new method in the EncryptedFile interface and keep the old method in ManifestListFile with a default impl that just forwards.
but I am also ok to keep the old name

Could there technically be custom implementations of ManifestListFile out there overriding encryptionKeyID to not delegate to the new function, that would break the design here, because internally we would only call the new version of the function?
I think it's cleaner to keep the same API that we had before.

I went through this again, and I think I managed to come up with a solution that isn't a breaking change for custom implementations of ManifestListFile. See EncryptionUtil.decryptKeyMetadata functions where I introduced a variation that accepts the key ID from the various versions.

gaborkaszab

Thanks for taking a look, @stevenzwu !

gaborkaszab · 2026-08-08T10:28:03Z

+ * A file that may be encrypted. If it is encrypted, its encrypted key metadata is tracked in the
+ * table metadata encryption keys and is referenced by a key ID.
+ */
+public interface EncryptableFile {


Changed to EncryptedFile, however, I'm still not comfortable with the vile name. The name suggests the file is encrypted, but it's may or may not. Maybe FileWithEncryptedKey ?

gaborkaszab · 2026-08-08T10:29:37Z

+ * The file key metadata can be encrypted. Returns ID of encryption key or null if it's not
+ * encrypted.
+ */
+ String encryptionKeyID();


Thanks for the comment suggestion! Applied.
About the function name: I'm not sure we can change the function name, because that would break the API for ManifestListFile that is derived from this interface.
Alternatively, we can avoid deriving ManifestListFile from the new interface, but then we won't be able to cast it to the new interface and delegate to the new function signature like in FileIO or in EncryptionUtil:
newInputFile((EncryptableFile) manifestList)

varun-lakhyani

Overall LGTM.

Data types / return types are consistently migrated to EncryptedFile.
Deprecation javadocs are clear
ManifestListFile's own deprecation javadoc + extends EncryptedFile explains most of the seemingly-confusing type/return changes across the PR (e.g. toManifestListFile() returning EncryptedFile).

anoopj · 2026-08-19T23:39:32Z

 * @param manifestList a ManifestListFile
 * @param em the table's EncryptionManager
 * @return a decrypted key metadata buffer
+ * @deprecated since 1.12.0. Will be removed in 1.13.0; use {@link


Shouldn't the removal version be the next major version (2.0.0) as you have done in the other deprecation tag in this PR?

In core/ we can remove stuff in the next minor release, while in api/ it's stricter, the next major is the closest one for removal.

Giving this a second thought I think you're right. As long as we keep the ManifestListFile interface (until 2.0.0 as it's in api/) we have to keep the encryption/decryption methods for that class. Otherwise it might be a behavior change for custom implementation if we silently fall back to the method for EncryptedFile.

anoopj · 2026-08-19T23:57:06Z

+ String encryptionKeyID();
+
+ /** Decrypt and return the file key metadata */
+ ByteBuffer decryptKeyMetadata(EncryptionManager em);


Carry over from existing design, but this method feels like it belongs on the encryption service rather than on the file value type. The file only supplies the key ID. I don't think it can simply be dropped, though: the caller (EncryptingFileIO) is in api and EncryptionUtil is in core, so api can't call it directly.
If we want it off EncryptedFile, the clean version is to move the operation onto EncryptionManager (impl already in core, and EncryptingFileIO already holds one), keyed by encryptionKeyID().
It's fine if we want to do in a followup.
cc @rdblue in case he has any thoughts.

Hmm, with this the only purpose of putting this functionality into EncryptionManager is that its implementation in core/ can delegate to EncryptionUtil to get around the issue that EncryptingFileIO.newInputFile() is in api/ and can't call the util method in core/. I'm not entirely comfortable with that design, while I'm not comfortable with the current one either. At least we keep existing api with the current one.

I likes Anoop suggestion here of dropping this method and let callers use the EncryptionUtil.decryptXXX directly. Since we are defining a new interface, it seems like a good opportunity to deprecate. Then this class becomes just a POJO envelope interface.

I gave this a try and in the latest version of this PR you can see how this can be implemented. I dropped the decryptKeyMetadata method from the new interface. However, to be able to use the underlying EncryptionUtil.decryptKeyMetadata() that is in core/, from EncryptingFileIO that is in api/, I had to introduce the same in EncryptionManager. Basically, this is inline with @anoopj 's suggestion. Let me know what you think!

gaborkaszab

Thanks for looking into this @stevenzwu @huaxingao @anoopj !
I renamed the new interface, also went for the simpler method keyId() while keeping compatibility with custom ManifestListFile implementations.

gaborkaszab · 2026-08-25T11:48:43Z

 * @param manifestList a ManifestListFile
 * @param em the table's EncryptionManager
 * @return a decrypted key metadata buffer
+ * @deprecated since 1.12.0. Will be removed in 1.13.0; use {@link


Giving this a second thought I think you're right. As long as we keep the ManifestListFile interface (until 2.0.0 as it's in api/) we have to keep the encryption/decryption methods for that class. Otherwise it might be a behavior change for custom implementation if we silently fall back to the method for EncryptedFile.

gaborkaszab · 2026-08-25T11:50:55Z

+ * The file key metadata can be encrypted. Returns ID of encryption key or null if it's not
+ * encrypted.
+ */
+ String encryptionKeyID();


I went through this again, and I think I managed to come up with a solution that isn't a breaking change for custom implementations of ManifestListFile. See EncryptionUtil.decryptKeyMetadata functions where I introduced a variation that accepts the key ID from the various versions.

gaborkaszab · 2026-08-25T12:13:53Z

+ String encryptionKeyID();
+
+ /** Decrypt and return the file key metadata */
+ ByteBuffer decryptKeyMetadata(EncryptionManager em);


Hmm, with this the only purpose of putting this functionality into EncryptionManager is that its implementation in core/ can delegate to EncryptionUtil to get around the issue that EncryptingFileIO.newInputFile() is in api/ and can't call the util method in core/. I'm not entirely comfortable with that design, while I'm not comfortable with the current one either. At least we keep existing api with the current one.

gaborkaszab · 2026-08-25T12:50:48Z

+ * A file that may be encrypted. If it is encrypted, its encrypted key metadata is tracked in the
+ * table metadata encryption keys and is referenced by a key ID.
+ */
+public interface EncryptableFile {


Renamed the interface to FileWithEncryptedKey

…ManifestListFile ManifestListFile and its implementation contains nothing that is specific to manifest lists. It is more generally related to files that use TableMetadata.encryptionKeys to store encrypted encryption key metadata that are referred to by a key ID. This PR introduces a more general interface that can be used accross multiple file types like manifest lists, V4 root manifests, table statistics and partition statistics. The less general functionality specific to manifest lists is deprecated or removed where possible.

stevenzwu · 2026-09-03T20:34:33Z

+ * A file that may be encrypted. If it is encrypted, its encrypted key metadata is tracked in the
+ * table metadata encryption keys and is referenced by a key ID.
+ */
+public interface FileWithEncryptedKey {


I know we have gone through a couple of iterations on naming (EncryptableFile, EncryptedFile), which is a bit tricky in this case.
Just throwing out a couple more for considerations: FileWithKeyId or FileWithEncryptionKeyId?

I like FileWithKeyId, renamed the new interface to that. Thanks for the suggestion @stevenzwu !

Also, one minor correction to ManifestListFile: We can't default encryptionKeyId() to return keyId, because custom implementations might not implement keyId(), hence would be broken.

github-actionsBot added API core labels Aug 7, 2026

This was referenced Aug 7, 2026
API, Core, Spark, Flink: extract SnapshotFile abstraction #17523
Open
API, Core, Spark 4.1: Add key_metadata and encrypted write support for StatisticsFile #17417
Open

gaborkaszab commented Aug 7, 2026
View reviewed changes

stevenzwu added this to V4: metadata tree Aug 7, 2026

stevenzwu moved this to In review in V4: metadata tree Aug 7, 2026

stevenzwu reviewed Aug 7, 2026
View reviewed changes

stevenzwu changed the title ~~API, Core: Refactor: Introduce a more general abstraction to replace ManifestListFile~~API, Core: Introduce a generic EncryptedFile abstraction to replace ManifestListFileAug 7, 2026

gaborkaszab force-pushed the main_encryptable_file branch from 0a3b903 to c08b3fdCompare August 8, 2026 10:50

gaborkaszab commented Aug 8, 2026
View reviewed changes

gaborkaszab requested a review from stevenzwu August 8, 2026 10:50

varun-lakhyani approved these changes Aug 10, 2026
View reviewed changes

stevenzwu approved these changes Aug 10, 2026
View reviewed changes

huaxingao reviewed Aug 14, 2026
View reviewed changes

Comment threadcore/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java Outdated

gaborkaszab changed the title ~~API, Core: Introduce a generic EncryptedFile abstraction to replace ManifestListFile~~API, Core: Introduce a generic abstraction to replace ManifestListFileAug 14, 2026

anoopj reviewed Aug 19, 2026
View reviewed changes

gaborkaszab mentioned this pull request Aug 24, 2026
Core: Add V4 scan task planner #17541
Open

gaborkaszab commented Aug 25, 2026
View reviewed changes

gaborkaszab added 2 commits August 25, 2026 14:58

Rename new interface, introduce keyId() instead of encryptionKeyID()
0f9f10f

gaborkaszab force-pushed the main_encryptable_file branch from 5ee1f33 to 1000d01Compare August 25, 2026 13:01

spotless check fixes
d3dec6d

gaborkaszab force-pushed the main_encryptable_file branch from 1000d01 to d3dec6dCompare August 25, 2026 13:20

stevenzwu reviewed Sep 3, 2026
View reviewed changes

Rename to FileWithKeyId, Remove decryptKeyMetadata() from interface
0b5203b
Also, one minor correction to ManifestListFile: We can't default encryptionKeyId() to return keyId, because custom implementations might not implement keyId(), hence would be broken.

gaborkaszab requested review from anoopj, huaxingao and stevenzwu September 7, 2026 15:40

Uh oh!

Conversation

gaborkaszab commented Aug 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwuAug 7, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwuAug 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

varun-lakhyani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anoopjAug 19, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

stevenzwuAug 7, 2026•
edited
Loading

stevenzwuAug 9, 2026•
edited
Loading

anoopjAug 19, 2026•
edited
Loading