Comet native writer crashes for non Arrow input data

[comphead]

Describe the bug

Run with

--conf spark.comet.parquet.write.enabled=true --conf spark.comet.operator.DataWritingCommandExec.allowIncompatible=true

the command

 spark.range(1000).write.mode("overwrite").parquet("/tmp/writertest/input")

it fails with

Caused by: org.apache.spark.SparkException: Comet execution only takes Arrow Arrays, but got class org.apache.spark.sql.execution.vectorized.OnHeapColumnVector

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[Shekharrajak]

Looks like this is coming from

datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeWriteExec.scala

Line 146 in 3a6452b

child.execute().mapPartitionsInternal { _ =>

As comment suggest, let me try converting Spark batches to Arrow format using CometArrowConverters.columnarBatchToArrowBatchIter()