Skip to content

Delete/Update fails for tables with more than 1000 columns #6368

Description

@ayushtkn

Apache Iceberg version

1.1.0 (latest release)

Query engine

Hive

Please describe the bug 🐞

In case the columns in the table are more than 1000, the update & delete operations on a V2 tables fails due to collision with the partition field id's which starts from 1000.

Caused by: java.lang.IllegalArgumentException: Multiple entries with same key: 1001=q581316 and 1001=_partition.month at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:376)
 at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:370)
 at org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:153)
 at org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:115)
 at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap$Builder.buildOrThrow(ImmutableMap.java:574)
 at org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:538)
 at org.apache.iceberg.types.IndexByName.byId(IndexByName.java:80)
 at org.apache.iceberg.types.TypeUtil.indexNameById(TypeUtil.java:161)
 at org.apache.iceberg.Schema.lazyIdToName(Schema.java:167)
 at org.apache.iceberg.Schema.<init>(Schema.java:108)
 at org.apache.iceberg.Schema.<init>(Schema.java:91)
 at org.apache.iceberg.Schema.<init>(Schema.java:83)
 at org.apache.iceberg.Schema.<init>(Schema.java:79)
 at org.apache.iceberg.mr.hive.IcebergAcidUtil.createFileReadSchemaWithVirtualColums(IcebergAcidUtil.java:89)
 at org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.readSchema(IcebergInputFormat.java:497)
 at org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.initialize(IcebergInputFormat.java:258)
 at org.apache.iceberg.mr.mapred.AbstractMapredIcebergRecordReader.<init>(AbstractMapredIcebergRecordReader.java:40)
 at org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.<init>(MapredIcebergInputFormat.java:89)
 at org.apache.iceberg.mr.mapred.MapredIcebergInputFormat.getRecordReader(MapredIcebergInputFormat.java:79)
 at org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getRecordReader(HiveIcebergInputFormat.java:170)
 at org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
 at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:461) ... 27 more ]]

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions