Skip to content

Track encoded file concatenation and column slices #8695

Description

@Xuanwo

Goal

Land a single low-level primitive for concatenating compatible encoded Lance files and FileFragment APIs for independently writing and combining complete physical-row slices of existing columns.

Context

Contract

  • Compatible current-version files concatenate encoded page payloads without decode and re-encode.
  • Unsupported layouts use the existing ordered decode/re-encode fallback; corrupt or missing inputs remain errors.
  • Column slices are immutable, fragment-local physical-row intervals with exact field and source-snapshot identity.
  • Publication requires complete, gap-free, non-overlapping fragment coverage.
  • Compaction reuses the same encoded-file primitive without adopting the column-slice lifecycle.
  • Python ColumnSlice values support a versioned bytes round trip and pickling.

Out of scope

  • Task planning, scheduling, retries, progress, recovery, or distributed commit coordination.
  • A new manifest layout or transaction operation.
  • Caller-owned staged-file retention and cleanup.

Completion gates

  • feat: add column slice stitching #8660 is merged.
  • Python serialization and fragment bindings implement the proposal contract.
  • Encoded concatenation, fallback, malformed-input, coverage, and compaction-reuse tests pass.
  • Public API and ownership boundaries are documented.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions