feat: make data file parts serializable - #9072
Merged
Merged
Conversation
Xuanwo
marked this pull request as ready for review
September 9, 2026 10:04
Contributor
There was a problem hiding this comment.
The serializable descriptor design reopens and validates staging files before assembly, with coordinator fencing and fragment row coverage explicitly owned by callers. The author accepts that finish does not reclaim Blob payloads from lost or unused retry leases, so a referenced target can retain those payloads indefinitely. No further change is requested for this pull request.
jackye1995
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Distributed writers need to persist a target and completed parts so another process can resume writing or assemble the final file.
DataFileTargetandDataFilePartnow support serde, and Lance creates staging files when writing parts. Assembly reopens and validates the referenced files before producing the finalDataFile.Managed Blob payloads are written directly into the final target's sidecar namespace and are not copied during assembly. Callers explicitly invoke
target.finish(dataset)after commit to remove staging parts, ortarget.cleanup(dataset)after abandoning the target to remove staging parts, the final file, and Blob payloads. Both operations tolerate missing objects and can be retried after partial cleanup. Callers remain responsible for stopping users of the target, retaining the same dataset/base mapping, allocating disjoint Blob ID ranges across retries, and coordinating checkpoints with dataset GC.This replaces the high-level caller-provided Writer and
open_partflow with serializable part descriptions. Callers useDataset::concat_data_file_partsand constructDataReplacementGroupthemselves; fragment row coverage and commit fencing remain their responsibility. No separateFileFragment::write_columns_from_partsentry point is retained.Checkpoint contents must be trusted, and callers must fence stale workers so they cannot resume writes or assembly after a target is committed. These descriptors do not prove ownership or consult retained manifests.
finishretains all Blob payloads under the target, including unused retry leases; ordinary dataset GC also preserves them while the parent file is referenced. Per-Blob reachability collection is outside this change.Follow-up to #8923.