Iceberg TestMetadataTablesWithPartitionEvolution.testPartitionColumnNamedPartition returns rows in a different order with native Iceberg writes enabled

[andygrove]

Describe the bug

With spark.comet.write.iceberg.splitOperator.enabled and spark.comet.iceberg.write.enabled both on (#5644, experiment PR #5677), the upstream Iceberg test TestMetadataTablesWithPartitionEvolution.testPartitionColumnNamedPartition fails because an unordered SELECT * returns its rows in a different order than iceberg-java produces:

org.opentest4j.AssertionFailedError: [Should return all expected rows: row 1 col 1 contents should match]
expected: 1
 but was: 3
    at org.apache.iceberg.spark.SparkTestHelperBase.assertEquals(SparkTestHelperBase.java:85)
    at org.apache.iceberg.spark.source.TestMetadataTablesWithPartitionEvolution.testPartitionColumnNamedPartition(TestMetadataTablesWithPartitionEvolution.java:652)

The test writes four rows across two partitions and compares the result of SELECT * positionally:

sql("CREATE TABLE %s (id int, partition int) USING iceberg PARTITIONED BY (partition)", tableName);
sql("INSERT INTO %s VALUES (1, 1), (2, 1), (3, 2), (2, 2)", tableName);
List<Object[]> expected = ImmutableList.of(row(1, 1), row(2, 1), row(3, 2), row(2, 2));
assertEquals("Should return all expected rows", expected, sql("SELECT * FROM %s", tableName));

The first row read back is (3, ...) rather than (1, 1), so the partition = 2 data file is coming back first.

Steps to reproduce

Run the Iceberg Spark SQL Tests jobs with both write flags defaulted on. Observed in run 33995765073 at commit d86d3b5:

• iceberg-spark/iceberg-1.11.0/spark-4.1.3: 5 parameter combinations fail (fileFormat = ORC/formatVersion 1, PARQUET/1, PARQUET/2, AVRO/1, AVRO/2, across the hadoop, hive and hive-with-parquet-enabled=false catalogs)
• iceberg-spark/iceberg-1.9.1/spark-3.5.9: 2 combinations fail (fileFormat = ORC, formatVersion 1 and 2, testhive catalog)
• The 1.10.0/3.5 and 1.8.1/3.4 jobs do not fail this test

Expected behavior

Either the write path produces the same data-file layout and read-back order as iceberg-java so the upstream assertion holds, or we establish that the assertion is order-dependent and adapt it in dev/diffs/iceberg/*.diff (never by disabling Comet).

Additional context

Needs triage before it can be fixed: the failing set includes ORC and AVRO file formats, where the native Parquet writer cannot be the one producing the files, and it covers only a subset of the parameter matrix while other versions pass the same test. That pattern is as consistent with an order-dependent assertion that Comet makes unstable (read-side task or file ordering) as with a writer file-layout divergence, so the first step is to determine which side changes the order.

Part of the native Iceberg writes epic, #5649, and a blocker for #5644. Surfaced by the failure inventory on #5677.