[GLUTEN-12538][VL] Fall back to row-based Delta stats when the stats plan cannot be offloaded

[felipepessoto]

What changes are proposed in this pull request?

Fixes#12538.

GlutenDeltaJobStatsTracker collects Delta write statistics by building a local Velox aggregation plan and assuming it is always offloaded to a WholeStageTransformer. When the statistics plan references a type or expression Velox cannot offload — for example an aggregate over a TIMESTAMP_NTZ column — the offload rules reject it and leave a vanilla ProjectExec, so the unconditional cast threw:

java.lang.ClassCastException: class org.apache.spark.sql.execution.ProjectExec cannot be cast to
class org.apache.gluten.execution.WholeStageTransformer
at ...GlutenDeltaJobStatsTracker$GlutenDeltaTaskStatsTracker.<init>(GlutenDeltaJobStatsTracker.scala:174)
at ...GlutenDeltaJobStatsTracker.newTaskInstance(GlutenDeltaJobStatsTracker.scala:80)

failing the whole write.

This PR makes the Delta stats path fall back to row-based statistics collection when the plan cannot be fully offloaded, as suggested by rui-mo and zhztheplayer in 12388:

• The decision is made on the executor in newTaskInstance(), where the native plan is actually built.
• tryBuildOffloadedStatsPlan builds the aggregation plan, runs the offload rules, and returns the offloaded TransformSupportonly when the whole plan was offloaded — every node except the StatisticsInputNode leaf is a TransformSupport. A root WholeStageTransformer alone is not sufficient, because ColumnarCollapseTransformStages can wrap an offloaded parent above a vanilla child.
• Otherwise it returns None and statistics are collected the row-based way through the existing GlutenDeltaJobStatsFallbackTracker. Native resources (the single-thread executor and NativePlanEvaluator) are allocated only on the accepted path.

Both the Delta 3.3 (src-delta33) and Delta 4.0+ (src-delta40) variants are updated.

How was this patch tested?

Added DeltaTimestampNtzStatsWriteSuite (both src-delta33 and src-delta40), which writes a Delta table with a top-level and a struct-nested TIMESTAMP_NTZ column through the native write path with statistics collection enabled. Without the fix the write throws the ClassCastException above; with it the write succeeds, the data reads back correctly, and Delta minValues/maxValues statistics are still produced for the TIMESTAMP_NTZ column via the fallback.

Compiles for -Pspark-4.1 (delta40) and -Pspark-3.5 (delta33).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI (Claude Opus 4.8)

[Smallfu666]

I independently reproduced this on Spark 4.0 and validated the row-based fallback. One question: could tryBuildOffloadedStatsPlan avoid catching all NonFatal exceptions and fall back only for explicit non-offload outcomes, so unexpected planner defects remain visible?

[felipepessoto]

I'll wait until #12388 is merged before publishing this and other PRs depending on it for validation, to avoid confusion.

[felipepessoto]

Instead of adding the fallback, I'm adding support for timestamp_ntz aggregation: #12967

[felipepessoto]

I independently reproduced this on Spark 4.0 and validated the row-based fallback. One question: could tryBuildOffloadedStatsPlan avoid catching all NonFatal exceptions and fall back only for explicit non-offload outcomes, so unexpected planner defects remain visible?

@Smallfu666, could you review these two PRs instead? #12967, #12966