Part of #5572.
map_from_arrays reports Incompatible whenever spark.sql.mapKeyDedupPolicy is LAST_WIN:
if (MapKeyDedupPolicySupport.isLastWin) {
Incompatible(Some(MapKeyDedupPolicySupport.incompatibleReason))
}spark/src/main/scala/org/apache/comet/serde/maps.scala:163-164
With no CodegenDispatchFallback mixin, that means the whole projection falls back to Spark for any session running that policy.
CometMapFromEntries sits directly below it in the same file and already carries the mixin (:211-213) for its own analogous restriction, so the two closely related functions behave differently for no good reason.
MapFromArrays.doGenCode builds the map through ArrayBasedMapBuilder, which is where the dedup policy is actually implemented, so the dispatcher gets LAST_WIN right by construction. MapType is supported by CometBatchKernelCodegen.isSupportedDataType recursively.
Fix: mix CodegenDispatchFallback into CometMapFromArrays.
Not in scope: the separate null-key divergence documented in getCompatibleNotes and tracked as #4680. That one is a Compatible path today, so it is a correctness question rather than a fallback question — though it is worth noting that routing through the dispatcher would fix it too, since Spark's builder raises "Cannot use null as map key" itself.
Part of #5572.
map_from_arraysreportsIncompatiblewheneverspark.sql.mapKeyDedupPolicyisLAST_WIN:spark/src/main/scala/org/apache/comet/serde/maps.scala:163-164With no
CodegenDispatchFallbackmixin, that means the whole projection falls back to Spark for any session running that policy.CometMapFromEntriessits directly below it in the same file and already carries the mixin (:211-213) for its own analogous restriction, so the two closely related functions behave differently for no good reason.MapFromArrays.doGenCodebuilds the map throughArrayBasedMapBuilder, which is where the dedup policy is actually implemented, so the dispatcher getsLAST_WINright by construction.MapTypeis supported byCometBatchKernelCodegen.isSupportedDataTyperecursively.Fix: mix
CodegenDispatchFallbackintoCometMapFromArrays.Not in scope: the separate null-key divergence documented in
getCompatibleNotesand tracked as #4680. That one is aCompatiblepath today, so it is a correctness question rather than a fallback question — though it is worth noting that routing through the dispatcher would fix it too, since Spark's builder raises "Cannot use null as map key" itself.