Skip to content

Core: Merge concurrently added DVs for RowDelta pure delete commits - #17754

Open
raunaqmorarka wants to merge 1 commit into
apache:mainfrom
raunaqmorarka:dv-conflict-merge
Open

Core: Merge concurrently added DVs for RowDelta pure delete commits#17754
raunaqmorarka wants to merge 1 commit into
apache:mainfrom
raunaqmorarka:dv-conflict-merge

Conversation

@raunaqmorarka

@raunaqmorarka raunaqmorarka commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Concurrent DELETEs on V3 tables fail with Found concurrently added DV when they add DVs for the same data file, even though position deletes are commutative.

This change merges the concurrently added DV into the operation's DV instead of failing, when all of the following hold:

  • the RowDelta does not add or remove data files
  • validateNoConflictingDeleteFiles is not enabled
  • the concurrent DV was added by a delete operation (a DV added by an overwrite marks rows that live on in rewritten data files, so merging it would drop this operation's deletes of those rows)

The newest concurrent DV for a data file contains the content of all older ones, since the V3 spec requires a DV to include all positions of the DV it replaces. That DV is removed from the table and its content is merged into the new DV, and pending removals of delete files for that data file are dropped. Merged puffin files from invalidated commit attempts are deleted on retry and on commit failure.

Comment thread core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java Outdated
Concurrent DELETE commits producing DVs for the same data file always
fail validation even though position deletes are commutative.
@sezruby

sezruby commented Sep 11, 2026

Copy link
Copy Markdown

Reviewed this end to end. The concurrent-DV merge is correct — the union preserves all deleted positions given the one-DV-per-file invariant, and restricting merges to DELETE-sourced snapshots (mergeableSnapshotIds) correctly excludes overwrite/replace-added DVs. Test coverage is comprehensive: retry idempotency, a concurrent DV replaced between attempts (with the prior merged file's deletion asserted), cleanup on commit failure, and the fail-closed cases for adds/removes-data-files and overwrite-sourced DVs.

@uros-b the FileIO-resolved cleanup-location comment looks addressed — would you mind another look? This is a clean base to extend row-level concurrency to UPDATE/MERGE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants