Removing partitionField and dropping partition columns will corrupt the entire table.

[Zhanxiao-Ma]

Apache Iceberg version

1.8.1

Query engine

None

Please describe the bug 🐞

I am using Iceberg through the Java API.
The table schema: a, b, c. Select column a as the partition column and write data. Call the interface Table.updateSpec().removeField() to remove the partition information, then call the interface Table.updateSchema().deleteColumn("a"). When subsequently attempting to read the table, an error occurs. In my expectation, this shouldn't cause an error, right?

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Zhanxiao-Ma]

Stack trace:
java.lang.NullPointerException: Type cannot be null

at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:922)
at org.apache.iceberg.types.Types$NestedField.<init>(Types.java:635)
at org.apache.iceberg.types.Types$NestedField.optional(Types.java:527)
at org.apache.iceberg.PartitionSpec.partitionType(PartitionSpec.java:134)
at org.apache.iceberg.expressions.ManifestEvaluator.<init>(ManifestEvaluator.java:67)
at org.apache.iceberg.expressions.ManifestEvaluator.forPartitionFilter(ManifestEvaluator.java:63)
at org.apache.iceberg.ManifestGroup.lambda$entries$8(ManifestGroup.java:251)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145)
at com.github.benmanes.caffeine.cache.UnboundedLocalCache.lambda$computeIfAbsent$2(UnboundedLocalCache.java:239)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at com.github.benmanes.caffeine.cache.UnboundedLocalCache.computeIfAbsent(UnboundedLocalCache.java:235)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:56)
at org.apache.iceberg.ManifestGroup.lambda$entries$9(ManifestGroup.java:274)
at org.apache.iceberg.io.CloseableIterable$5.shouldKeep(CloseableIterable.java:156)
at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:66)
at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49)
at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:64)
at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49)
at org.apache.iceberg.io.CloseableIterator$3.hasNext(CloseableIterator.java:91)
at org.apache.iceberg.relocated.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at org.apache.iceberg.io.CloseableIterable$ConcatCloseableIterable$ConcatCloseableIterator.hasNext(CloseableIterable.java:264)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)

[Zhanxiao-Ma]

call method：
table.newScan().planFiles().iterator().forEachRemaining(fileScanTask -> {
dataFiles.add(fileScanTask.file());
});

[nastra]

@Zhanxiao-Ma can you check whether this still fails with Iceberg 1.10.0? This should have been fixed by #11868

[Zhanxiao-Ma]

@Zhanxiao-Ma can you check whether this still fails with Iceberg 1.10.0? This should have been fixed by #11868
I used the code from the main branch to conduct relevant tests. The issue described in the GitHub issue has been resolved, but a new problem was found.

Table Schema: a, b, c

1 Select column a as the partition column and write data (DataFile1 with partition info).
2 Call the interface Table.updateSpec().removeField() to remove the partition information.
3 Then call the interface Table.updateSchema().deleteColumn("a").
4 Execute Table.newDelete().deleteFile(DataFile1).commit(), but it throws an error:
Cannot determine comparator for type: unknown
java.lang.UnsupportedOperationException: Cannot determine comparator for type: unknown
at org.apache.iceberg.types.Comparators.forType(Comparators.java:76)
at org.apache.iceberg.types.Comparators.internal(Comparators.java:82)
at org.apache.iceberg.types.Comparators$StructLikeComparator.lambda$new$0(Comparators.java:104)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Collections$2.tryAdvance(Collections.java:4747)
at java.base/java.util.Collections$2.forEachRemaining(Collections.java:4755)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
at org.apache.iceberg.types.Comparators$StructLikeComparator.(Comparators.java:106)
at org.apache.iceberg.types.Comparators.forType(Comparators.java:54)
at org.apache.iceberg.util.StructLikeWrapper.forType(StructLikeWrapper.java:36)
at org.apache.iceberg.util.StructLikeSet.lambda$new$0(StructLikeSet.java:44)
at java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
at org.apache.iceberg.util.StructLikeSet.add(StructLikeSet.java:102)
at org.apache.iceberg.util.StructLikeSet.add(StructLikeSet.java:32)
at org.apache.iceberg.util.PartitionSet.add(PartitionSet.java:93)
at org.apache.iceberg.ManifestFilterManager.delete(ManifestFilterManager.java:183)
at org.apache.iceberg.MergingSnapshotProducer.delete(MergingSnapshotProducer.java:197)
at org.apache.iceberg.StreamingDelete.deleteFile(StreamingDelete.java:49)
at org.apache.iceberg.StreamingDelete.deleteFile(StreamingDelete.java:24)
at org.apache.iceberg.TestOverwrite.testOverwriteFailedPartition(TestOverwrite.java:219)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)

[Zhanxiao-Ma]

@Zhanxiao-Ma can you check whether this still fails with Iceberg 1.10.0? This should have been fixed by #11868

The current support for UnknownType in Iceberg seems to be incomplete.

@Zhanxiao-Ma can you check whether this still fails with Iceberg 1.10.0? This should have been fixed by #11868
I used the code from the main branch to conduct relevant tests. The issue described in the GitHub issue has been resolved, but a new problem was found.

Table Schema: a, b, c

1 Select column a as the partition column and write data (DataFile1 with partition info). 2 Call the interface Table.updateSpec().removeField() to remove the partition information. 3 Then call the interface Table.updateSchema().deleteColumn("a"). 4 Execute Table.newDelete().deleteFile(DataFile1).commit(), but it throws an error: Cannot determine comparator for type: unknown java.lang.UnsupportedOperationException: Cannot determine comparator for type: unknown at org.apache.iceberg.types.Comparators.forType(Comparators.java:76) at org.apache.iceberg.types.Comparators.internal(Comparators.java:82) at org.apache.iceberg.types.Comparators$StructLikeComparator.lambda$new$0(Comparators.java:104) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Collections$2.tryAdvance(Collections.java:4747) at java.base/java.util.Collections$2.forEachRemaining(Collections.java:4755) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550) at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517) at org.apache.iceberg.types.Comparators$StructLikeComparator.(Comparators.java:106) at org.apache.iceberg.types.Comparators.forType(Comparators.java:54) at org.apache.iceberg.util.StructLikeWrapper.forType(StructLikeWrapper.java:36) at org.apache.iceberg.util.StructLikeSet.lambda$new$0(StructLikeSet.java:44) at java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305) at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195) at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172) at org.apache.iceberg.util.StructLikeSet.add(StructLikeSet.java:102) at org.apache.iceberg.util.StructLikeSet.add(StructLikeSet.java:32) at org.apache.iceberg.util.PartitionSet.add(PartitionSet.java:93) at org.apache.iceberg.ManifestFilterManager.delete(ManifestFilterManager.java:183) at org.apache.iceberg.MergingSnapshotProducer.delete(MergingSnapshotProducer.java:197) at org.apache.iceberg.StreamingDelete.deleteFile(StreamingDelete.java:49) at org.apache.iceberg.StreamingDelete.deleteFile(StreamingDelete.java:24) at org.apache.iceberg.TestOverwrite.testOverwriteFailedPartition(TestOverwrite.java:219) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.Optional.ifPresent(Optional.java:183) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)

The current support for UnknownType in Iceberg seems to be incomplete. Do you have any suggestions to complete this operation? @nastra

[Zhanxiao-Ma]

Should I create a new issue to track the new bug mentioned above? @nastra

[nastra]

@Zhanxiao-Ma yes please open a new issue and attach any reproducible steps that you have (e.g. a small unit test)

[Zhanxiao-Ma]

@Zhanxiao-Ma yes please open a new issue and attach any reproducible steps that you have (e.g. a small unit test)

Thank you for your answer. I've already submitted a new issue.
#14343