Uh oh!
There was an error while loading. Please reload this page.
API, Spark: Support cleanup level in ExpireSnapshots action and procedure - #17929
Open
xanderbailey wants to merge 2 commits into
Open
API, Spark: Support cleanup level in ExpireSnapshots action and procedure#17929xanderbailey wants to merge 2 commits into
xanderbailey wants to merge 2 commits into
Conversation
Exposes the CleanupLevel introduced in apache#14287 through the Spark action and the expire_snapshots procedure. The Spark action always commits the core expiration with CleanupLevel.NONE and determines files to delete itself, so the level is applied to the file diff rather than passed through: ALL unchanged, deletes metadata and content files METADATA_ONLY deletes manifests, manifest lists and statistics files, and skips listing content files entirely so the manifests of expired snapshots are never read NONE commits the expiration and deletes nothing Also adds CleanupLevel.fromString so the procedure can accept the level as a string, mirroring DeleteOrphanFiles.PrefixMismatchMode.
…rocedure Applies the same change to the remaining supported Spark versions. The action was byte-identical across all three versions, so it is copied verbatim. The 3.5 procedure differs structurally, but the parameter plumbing is the same.
xanderbailey
marked this pull request as ready for review
September 8, 2026 09:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CleanupLevelwas added to the expire snapshots interface in #14287 but is not reachable from any engine. Spark users cannot ask for metadata-only cleanup, and the only way to expire snapshots without deleting files is the core Java API.Exposing it is not a pass-through.
ExpireSnapshotsSparkActionalready commits the core expiration withCleanupLevel.NONEand works out what to delete itself, by anti-joining the file lists before and after expiration, so forwarding the level to core would have no effect. The level is applied to that diff instead:ALLbehaves as before,METADATA_ONLYdeletes manifests, manifest lists and statistics files but retains content files, andNONEcommits the expiration and deletes nothing.METADATA_ONLYdropscontentFileDSfrom both sides of the anti-join rather than filtering the result, so the manifests of expired snapshots are never read.Flink remains untouched in this PR