Uh oh!
There was an error while loading. Please reload this page.
Core, Data, Spark, Flink: Remove of position delete files with row data - #17706
Conversation
d39351b to
d48f943CompareThere was a problem hiding this comment.
Nit: flink/{v1.20,v2.0,v2.1}/FlinkAppenderFactory.java is removed entirely, but the PR body describes it as modified ("newPosDeleteWriter now builds path+pos writers"). The deletion is legitimate, the class is @deprecated since 1.11.0, will be removed in favor of FlinkFileWriterFactory, and is unreferenced in main, but its full removal spans a broader migration than PDWR removal alone - please confirm the scope and possibly correct the PR body description accordingly.
Uh oh!
There was an error while loading. Please reload this page.
| .writeProperties(writeProperties) | ||
| .build(); | ||
| SparkFileWriterFactory writerFactoryWithoutRow = | ||
| SparkFileWriterFactory writerFactory = |
There was a problem hiding this comment.
Could we add to the class javadoc, that if there is row-data in the position deletes then it will be removed after the rewrite?
There was a problem hiding this comment.
Same for the other Spark versions
There was a problem hiding this comment.
yeah I think it make sense, both position delete rewrite and table path rewrite will need to deal to existing PDWR. Given that we are removing all the basic writer in parquet I opted to throw exception and fail loudly instead. The expectation would be handle the PDWR conversion outside instead of silently drop the row in existing PD. Wondering if this align with your expectation ?
dramaticlly
commented
Aug 19, 2026
Convert to draft for now, as current change in rewrite-position-delete and rewrite-table-path direct drop the row for existing position delete with row data, need to think of a better way to handle the writer deletion while not cause trouble for existing workload. Maybe both need to fail early if detected existing PDWR |
1ad0d08 to
6eb042bComparePosition deletes that carry row data were deprecated in 1.11.0 for removal in 1.12.0, and the spec documents them as unsupported. Following the approach prototyped in apache#13870, the writers reject row data rather than silently dropping it. Writes are rejected; reads are preserved for backward compatibility with delete files produced by earlier versions. Generated-by: Claude Code
6eb042b to
dc5eec5Compare…data to 1.13.0 The removal of the ability to write position deletes with row data (PDWR) was scheduled for 1.12.0 (see apache#17706). The 1.12.0 release candidate is being cut now and the removal touches public API across Core, Data and Spark 3.5/4.0/4.1, so it should not be rushed through review. This moves the removal target of those deprecations from 1.12.0 to 1.13.0. The deprecations themselves are unchanged: everything is still deprecated as of 1.11.0 and callers should still migrate off the row-carrying overloads. Only the stated removal release moves out by one cycle. This mirrors apache#14392, which retargeted a batch of removal versions in the other direction. Markers moved to 1.13.0: - PositionDelete.set(CharSequence, long, R) and row() - PositionDeleteWriter constructor (appender type narrowing) - RewriteTablePathUtil.PositionDeleteReaderWriter.writer(..., Schema) - GenericFileWriterFactory positionDeleteRowSchema constructors, the builder setter, and the nine configureDataWrite/configureEqualityDelete/ configurePositionDelete methods superseded by FormatModelRegistry, whose removal was bundled into the same change - SparkFileWriterFactory positionDeleteRowSchema/positionDeleteSparkType builder setters, the row-schema constructor, and the runtime warning message - SparkPositionDeltaWrite.Context.deleteSparkType() Additionally, PositionDeltaWriter.delete(CharSequence, long, T, PartitionSpec, StructLike) is now deprecated for removal in 1.13.0. It was previously not deprecated at all, so without this its deprecation cycle would only start in 1.13.0 and the row parameter could not be dropped until 1.14.0. Deprecating it here keeps that narrowing on schedule for 1.13.0 alongside the rest. The annotation is additive: the overload stays abstract, so implementations are unaffected, while callers can migrate to the four-argument overload today. All three SparkPositionDeltaWrite call sites already use it; the only override is BasePositionDeltaWriter, which already calls the deprecated PositionDelete.set(CharSequence, long, R). Flink is not deferred. Its only PDWR surface was FlinkAppenderFactory, whose entire position delete path is built on row data. That class was deprecated as of 1.11.0 in favor of FlinkFileWriterFactory for removal in 1.12.0, and nothing in the repo references it outside its own test, so it is removed here on schedule from all three Flink versions rather than carried into 1.13.0. That also drops org.apache.iceberg.io.TestAppenderFactory from iceberg-data: the three TestFlinkAppenderFactory classes were its only subclasses, so with those gone the abstract base exercises nothing. iceberg-flink is not a revapi project and TestAppenderFactory is a test class, so neither removal needs a revapi baseline entry. The REST scan-planning markers in BaseScanTaskResponse, PlanTableScanRequest and PlanTableScanResponse are left alone; they are handled separately in apache#17638.
Uh oh!
There was an error while loading. Please reload this page.
pvary
commented
Aug 27, 2026
Merged to main. |
dramaticlly
commented
Aug 27, 2026
Appreciate your thoughtful review, I will update the dev list to close this out. Thanks again @pvary |
Follow up removal of #14045, this removes ability of writing Position Deletes With Row (PDWR) in java reference implementation. https://iceberg.apache.org/spec/#position-delete-files-with-row-data
After the change, the Java implementation will no longer write PDWR and reading of existing position delete continue to work.
Also sent an email to dev@iceberg.apache.orghttps://lists.apache.org/thread/3tz63wok59hhqphswovvl9z0yrh0z7pl For 2 maintenance action that read existing and produce new position deletes now fail loudly with PDWR
IllegalArgumentExceptionwhen executerewrite_table_pathon a table containing such PDWRIllegalArgumentExceptionwhen executerewrite_position_delete_fileson a table containing such PDWRCore
PositionDelete.set(CharSequence, long, R)and getter ofrow()removed while keep thesize() == 3and the positionalget/setso that delete files already containing row data still read back correctly.Avro,ORCandParquetbuildPositionWriter()now throwUnsupportedOperationExceptionwhen arowSchemais set.RewriteTablePathUtil.PositionDeleteReaderWriter: therowSchemaoverload is gone and the 4-argwriter()is now abstract.GenericFileWriterFactoryno longer configures a position delete row schemaPositionDeltaWriter.delete(CharSequence, long, T, PartitionSpec, StructLike)is now deprecated for removal in 1.13.0. Itsrowargument is ignored. Callers should move todelete(CharSequence, long, PartitionSpec, StructLike), which will become the method implementations override.Engine changes
Applied uniformly across Spark 3.5 / 4.0 / 4.1 and Flink 1.20 / 2.0 / 2.1
Spark
Applied identically to Spark 3.5, 4.0 and 4.1.
SparkFileWriterFactoryno longer accepts a position delete row schema or Spark type, and the deprecated constructor and thenewPositionDeleteWriteroverride that existed only for the row schema are gone.SparkPositionDeltaWriteno longer threads a position delete Spark type through, and the deprecateddeleteSparkType()accessor is removed.SparkPositionDeletesRewriteuses a single writer factory and refuses row carrying deletes, as described above.RewriteTablePathSparkAction: SparkPositionDeleteReaderWriter implements the 4-arg writer().Flink
Applied identically to Flink 1.20, 2.0 and 2.1.
FlinkAppenderFactoryis removed. Its position delete path was built entirely on row data, and it was already deprecated for removal in 1.12.0.TestFlinkAppenderFactorygoes with it, and test helpers that took aFileAppenderFactorynow build their delete files directly.position_deletesmetadata tableThe table keeps its
rowcolumn so that PDWR written before 1.12.0 still read back or null otherwiseExtra Callout
d39351b include change for
PositionDeltaWriter.delete(CharSequence, long, T, PartitionSpec, StructLike)still takes arowparameter that is now ignored byBasePositionDeltaWriter. It is not deprecated, so narrowing it belongs in a 1.13.0 deprecation cycle.AI Disclosure
Model: Claude Opus 5 (1M context)
Platform/Tool: Claude Code
Human Oversight: reviewed
Prompt Summary: split #16449 into smaller self-contained PRs; verify each group compiles and tests green standalone
@pvary if you can help take a look at the change?
CC @szehon-ho for changes relate to position delete metadata table and PDWR in rewrite table path changes