Skip to content

feat: make data file parts serializable - #9072

Merged
jackye1995 merged 2 commits into
mainfrom
xuanwo/serializable-data-file-parts
Sep 9, 2026
Merged

feat: make data file parts serializable#9072
jackye1995 merged 2 commits into
mainfrom
xuanwo/serializable-data-file-parts

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 9, 2026

Copy link
Copy Markdown
Member

Distributed writers need to persist a target and completed parts so another process can resume writing or assemble the final file. DataFileTarget and DataFilePart now support serde, and Lance creates staging files when writing parts. Assembly reopens and validates the referenced files before producing the final DataFile.

Managed Blob payloads are written directly into the final target's sidecar namespace and are not copied during assembly. Callers explicitly invoke target.finish(dataset) after commit to remove staging parts, or target.cleanup(dataset) after abandoning the target to remove staging parts, the final file, and Blob payloads. Both operations tolerate missing objects and can be retried after partial cleanup. Callers remain responsible for stopping users of the target, retaining the same dataset/base mapping, allocating disjoint Blob ID ranges across retries, and coordinating checkpoints with dataset GC.

This replaces the high-level caller-provided Writer and open_part flow with serializable part descriptions. Callers use Dataset::concat_data_file_parts and construct DataReplacementGroup themselves; fragment row coverage and commit fencing remain their responsibility. No separate FileFragment::write_columns_from_parts entry point is retained.

Checkpoint contents must be trusted, and callers must fence stale workers so they cannot resume writes or assembly after a target is committed. These descriptors do not prove ownership or consult retained manifests. finish retains all Blob payloads under the target, including unused retry leases; ordinary dataset GC also preserves them while the parent file is referenced. Per-Blob reachability collection is outside this change.

Follow-up to #8923.

@github-actions github-actions Bot added the enhancement New feature or request label Sep 9, 2026
@Xuanwo
Xuanwo marked this pull request as ready for review September 9, 2026 10:04
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 9, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 9, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The serializable descriptor design reopens and validates staging files before assembly, with coordinator fencing and fragment row coverage explicitly owned by callers. The author accepts that finish does not reclaim Blob payloads from lost or unused retry leases, so a referenced target can retain those payloads indefinitely. No further change is requested for this pull request.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 9, 2026
@jackye1995
jackye1995 merged commit ebba581 into main Sep 9, 2026
41 of 42 checks passed
@jackye1995
jackye1995 deleted the xuanwo/serializable-data-file-parts branch September 9, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants