test: enable the Iceberg split-operator and native write by default to surface test failures

[andygrove]

Which issue does this PR close?

None, this is exploratory. Related to #5308 (native Iceberg write) and the split-operator work in #4658.

Draft, not for merge. The point is to find out what breaks.

Rationale for this change

spark.comet.iceberg.write.enabled has defaulted to false since it landed, so the iceberg-rust writer has
only ever run in the handful of Comet suites that set it explicitly: CometIcebergWriteActionSuite,
CometIcebergWriteDetectionSuite, CometIcebergRewriteActionSuite, and CometIcebergSystemFunctionSuite.
Everything else, including the Iceberg Spark SQL suites that run Iceberg's own tests against Comet, has been
writing through Spark's stock executor-side writer. So we do not currently know how much of the Iceberg test
surface the native writer actually passes.

spark.comet.write.iceberg.splitOperator.enabled is a different story. The Iceberg test diffs have set it to
true in every Comet-configured session since #5259, so the two-operator plan has been running against
Iceberg's write, commit, and row-level-operation tests all along. Only the native writer is genuinely new
here, and flipping the user-facing default keeps the two consistent.

Turning the native write flag on is not sufficient on its own. CometIcebergNativeWrite sets
requiresNativeChildren, and Spark emits a row-based LocalTableScanExec for an inline VALUES list, so a
write fed that way is declined and silently runs on the JVM writer. Iceberg's suites seed data with
INSERT INTO ... VALUES and df.writeTo(...).append() constantly, and nothing in those suites asserts which
writer produced the files, so without spark.comet.exec.localTableScan.enabled the native writer would stay
hidden from a large part of the write surface while everything still passed green.

Measured locally on Spark 4.1.3 with the Iceberg 1.11 runtime, using the session config the Iceberg diffs
install:

Write shape	Native write on	Native write plus local table scan
INSERT INTO ... VALUES, unpartitioned	JVM writer	native writer
df.writeTo(...).append(), unpartitioned	JVM writer	native writer

Writes that already had a Comet-native child reached the native writer either way: partitioned
INSERT INTO ... VALUES, INSERT INTO ... SELECT, copy-on-write UPDATE, CTAS, and a target carrying
write.parquet.compression-codec.

What changes are included in this PR?

• CometConf.scala: default spark.comet.write.iceberg.splitOperator.enabled and
  spark.comet.iceberg.write.enabled to true, and drop the now false "Off by default" sentence from the
  native write config doc.
• dev/diffs/iceberg/{1.8.1,1.9.1,1.10.0,1.11.0}.diff: set spark.comet.exec.localTableScan.enabled=true
  alongside the existing Comet configuration in every Comet-configured session, so VALUES driven and
  DataFrame driven writes can reach the native writer instead of silently falling back. Each diff was
  regenerated against its own tag following docs/source/contributor-guide/iceberg-spark-tests.md.
• docs/source/contributor-guide/iceberg-spark-tests.md: document that flag and the existing
  spark.comet.explainFallback.enabled setting, and drop the stale claim that the split-operator flag is off
  by default.

Deliberately not included: docs/source/user-guide/latest/iceberg-writes.md still says both features are
experimental and off by default, and still lists splitOperator.enabled being false as the first fallback
reason. If any of this is worth keeping, the guide has to be rewritten, but that is premature until we see the
failures.

How are these changes tested?

By CI, which is the whole point. The run-iceberg-tests label is applied so the Iceberg 1.8 / 1.9 / 1.10 jobs
run alongside 1.11, giving coverage across Spark 3.4 / 3.5 / 4.1 and four Iceberg versions. Comet's own
Iceberg suites run in the [scans] bucket on every profile regardless.

The Iceberg diffs also set spark.comet.explainFallback.enabled=true, so Comet logs every operator it
declines together with the reason. That output is not usable from CI as things stand: gradle does not forward
Spark's log output into the job log, and a 130k line Iceberg job log contains zero WARN lines. Reading fallback
rates needs either a gradle test-logging change or a local run. Until then, the pass or fail result is all CI
gives us, and it does not distinguish a native write from a silent fallback.

The regenerated diffs were checked to apply cleanly against their tags and to produce one
localTableScan.enabled line per existing splitOperator.enabled line. The measurement in the table above
came from a throwaway probe suite built on the same session config the diffs install.

What the run found

All eight Iceberg jobs fail. The failures are dominated by one bug, filed as #5689: converting
IcebergWriteExec to CometIcebergWriteExec drops the ColumnarToRow that sat between Iceberg's columnar
BatchScan and the whole-stage codegen stage above it, so every copy-on-write DELETE, UPDATE and MERGE dies
with ColumnarBatch cannot be cast to InternalRow. It only shows up when AQE is disabled, which is why no
existing Comet suite caught it. Isolated locally: the split-operator plan alone does not trigger it and the
local table scan flag is irrelevant, it needs the native write flag plus AQE off.

Beware of reading the extension-suite failure counts as a trend. Iceberg's ExtensionsTestBase sets
ADAPTIVE_EXECUTION_ENABLED to RANDOM.nextBoolean() per session, so the number of tests #5689 takes down
swings between runs of the same behaviour (99 in one run, 31 in the next).

Also found:

• Native Iceberg writer does not honour write.target-file-size-bytes the same way as iceberg-java #5690: the native writer rolls files at a different point than iceberg-java for the same
  write.target-file-size-bytes, producing consistently half as many files, which also perturbs compaction
  results in TestRewriteDataFilesAction.
• Native Iceberg write panics in construct_partition_summaries after partition spec evolution #5691: a panic in ManifestWriter::construct_partition_summaries (zip_eq length mismatch) when writing to
  a table whose partition spec has been evolved.
• Native Iceberg write fails with "No column with source column id" after an identity partition field is dropped #5693: No column with source column id 2 in schema after an identity partition field and its source column
  are dropped. Same phase as Native Iceberg write panics in construct_partition_summaries after partition spec evolution #5691 and possibly the same root cause.
• Native Iceberg write panics generating the partition path for a timestamptz partition column #5694: an Option::unwrap() panic in microseconds_to_datetimetz while building the partition path for a
  timestamptz partition column.
• Native Iceberg writer raises a different exception than iceberg-java for unclustered input to a clustered writer #5698: TestRequiredDistributionAndOrdering.testDisabledDistributionAndOrdering expects
  IllegalStateException from the JVM clustered writer and gets CometNativeException: The input is not sorted! instead. Same semantics, different exception type. This one is deterministic on all four Iceberg
  versions and passed before the local table scan flag was added, because the write used to fall back to the
  JVM writer.

In the [scans] jobs, several CometIcebergWriteActionSuite cases fail with
expected >= 1 IcebergWriteExec in captured plans, got 0. Those tests assert that the JVM split writer is
present in the plan, and the native writer now takes over instead. Their premise has to be revisited if these
defaults ever ship.

[andygrove]

CI failure baseline: d86d3b5 (run 33995765073)

97 failing tests across 13 jobs (8 Iceberg SQL jobs + 5 PR Build [scans] jobs).

Count	Cause	Suites
34	Native writer throws CometNativeException: Unexpected => The input is not sorted! Cannot write to partition that was previously closed where Iceberg expects IllegalStateException	TestRequiredDistributionAndOrdering (32), TestMetadataTablesWithPartitionEvolution (2)
28	Target file size not honored: writer emits half the expected data files (8 -> 4, 4 -> 2)	TestSparkDataWrite (24), TestRewriteDataFilesAction (4)
11	Plan assertions expect the two-op IcebergWriteExec, plan now has CometIcebergWrite	CometIcebergWriteActionSuite (same 11 in all 5 [scans] jobs)
8	Native panic: called Option::unwrap() on a None value	TestSparkDataFile
5	Wrong query results after partition evolution (expected: 1 but was: 3)	TestMetadataTablesWithPartitionEvolution
5	Native panic: itertools: .zip_eq() reached end of one iterator before the other	TestAlterTablePartitionFields
5	Invalid Iceberg scan task: DataInvalid => Non-empty FileScanTask partition requires a partition spec	TestForwardCompatibility (3), TestRewriteDataFilesAction (2)
1	Unexpected => No column with source column id 2 in schema	TestAlterTablePartitionFields

Per-job Iceberg counts: 1.8/3.4 spark 14, extensions 4; 1.9/3.5 spark 16, extensions 4; 1.10/3.5 spark 15, extensions 6; 1.11/4.1 spark 19, extensions 8.

Counts are test-case instances summed over jobs, so the same root cause is counted once per Iceberg version and once per Spark profile. Will repost this summary as fixes land on the branch.

Correction to the table above. Two rows were mis-grouped by the log parser (a stack trace bled into the next test's window). Corrected split, same 86 total:

• "The input is not sorted!" is 32, all TestRequiredDistributionAndOrdering. The 2 TestMetadataTablesWithPartitionEvolution cases attributed to it are actually row-order divergences.
• Wrong data-file counts is 30: TestSparkDataWrite 24 + TestRewriteDataFilesAction 6 (both testBinPackCombineMediumFiles and testRemoveDangledPositionDeletesPartitionEvolution are count assertions, not native errors). Noted on Native Iceberg writer does not honour write.target-file-size-bytes the same way as iceberg-java #5690.
• Row-order divergence is 7, all testPartitionColumnNamedPartition. Filed as Iceberg TestMetadataTablesWithPartitionEvolution.testPartitionColumnNamedPartition returns rows in a different order with native Iceberg writes enabled #5776.
• Non-empty FileScanTask partition requires a partition spec is 3, all TestForwardCompatibility. Fixed by fix: read Iceberg tables partitioned by an unknown transform #5759 (Iceberg native scan fails on a table partitioned by an unknown transform (TestForwardCompatibility CI failure) #5758).

Issue coverage for the eight causes: #5698 and #5690 open; #5691, #5693, #5694 (all fixed by #5729) and #5758 (fixed by #5759) closed and now on main; #5776 newly filed; the 11 CometIcebergWriteActionSuite plan assertions are test staleness rather than a bug and belong on the #5644 checklist.

This branch has since merged main (f15d424), which picks up #5729 and #5759, so the next run should clear the 16 native-panic and scan failures. Next summary will come from that run.