[GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation

[felipepessoto]

What changes are proposed in this pull request?

Enable native TIMESTAMP_NTZ aggregation in the Velox backend:

• Allow aggregate and shuffle operators, plus the projection expressions used by Delta statistics collection, through the TIMESTAMP_NTZ fallback validator.
• Accept TimestampNTZType aggregate buffers.
• Preserve TIMESTAMP_NTZ as a distinct tsntz function-signature type and as Substrait PrecisionTimestamp.
• Add native min/max and Delta statistics regression coverage, including nested TIMESTAMP_NTZ columns.
• Remove 42 Delta data-skipping tests fixed by the native statistics aggregation support from the known-failure baseline.

This prevents Delta's local statistics SortAggregateExec from falling back to a vanilla Spark plan and addresses the TIMESTAMP_NTZ failure described in #12538. This change is independent of #12966.

How was this patch tested?

• cpp/build/velox/tests/velox_plan_conversion_test --gtest_filter='VeloxSubstraitSignatureTest.*:VeloxToSubstraitTypeTest.*:VeloxSubstraitRoundTripTest.minMaxTimestampUtc'
• ./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/" -DwildcardSuites=org.apache.spark.sql.GlutenTimestampNtzAggregateSuite
• ./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/" -DwildcardSuites=org.apache.spark.sql.delta.GlutenDeltaStatsSuite
• ./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-3.5 -Pscala-2.12 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" -DwildcardSuites=org.apache.spark.sql.delta.GlutenDeltaStatsSuite
• ./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-3.5 -Pscala-2.12 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" -DwildcardSuites=org.apache.gluten.functions.DateFunctionsValidateSuite

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI 1.0.82

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

🔄 Delta Spark UT started by @felipepessoto (~2.5 h). View run

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

🟡 Changes recommended

A newly added Velox round-trip test uses millisecond-precision timestamps despite declaring microsecond precision, weakening coverage for the intended NTZ precision behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Enables native handling of TIMESTAMP_NTZ (Spark) / TIMESTAMP_UTC (Velox) in aggregation and related Delta statistics collection paths for the Velox backend, reducing Spark fallbacks and unblocking Delta stats plans previously rejected due to NTZ types.

Changes:

• Extend the TimestampNTZ fallback validator to allow aggregates, shuffles, and direct NTZ projections needed by Delta stats plans.
• Accept TimestampNTZType in aggregation buffer type checks and preserve a distinct native signature token (tsntz) plus Substrait PrecisionTimestamp encoding.
• Add native regression coverage (Spark UT + Velox C++ tests) and remove now-fixed Delta data-skipping cases from the known-failure baseline.

File summaries

File	Description
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala	Adds Spark 4.1 regression coverage for NTZ min/max aggregation and a projection fallback case.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Enables the new Spark 4.1 UT suite in Velox test settings.
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala	Broadens NTZ fallback validation to permit additional plan nodes/expressions used by native stats aggregation.
gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala	Maps Spark TimestampNTZType to Substrait timestamp NTZ type node and uses tsntz in signature naming.
gluten-substrait/src/main/scala/org/apache/gluten/execution/HashAggregateExecBaseTransformer.scala	Allows TimestampNTZType in supported aggregation buffer type checks.
cpp/velox/tests/VeloxToSubstraitTypeTest.cc	Adds a type-conversion test for TIMESTAMP_UTC -> Substrait PrecisionTimestamp.
cpp/velox/tests/VeloxSubstraitSignatureTest.cc	Adds tsntz signature coverage for TIMESTAMP_UTC mapping in both directions.
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc	Adds a min/max aggregation round-trip test over TIMESTAMP_UTC.
cpp/velox/substrait/VeloxToSubstraitType.cc	Encodes TIMESTAMP_UTC as Substrait precision_timestamp(6).
cpp/velox/substrait/VeloxSubstraitSignature.cc	Maps TIMESTAMP_UTC to/from the tsntz signature token.
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala	Adds Delta 4.0 stats regression coverage including nested NTZ columns.
backends-velox/src-delta33/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala	Adds Delta 3.3 stats regression coverage including nested NTZ columns.
.github/workflows/util/delta-spark-ut/known-failures.txt	Removes Delta data-skipping known failures resolved by native NTZ stats aggregation support.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[felipepessoto]

@Mariamalmesfer@rui-mo I think you two have worked on other timestamp_ntz PRs. Could you take a look, please?

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:1

• The previous implementation used dataType.typeName == "timestamp_ntz", which is resilient across Spark versions/shims. Switching to a direct TimestampNTZType reference introduces a compile-time dependency that can break builds for Spark variants where TimestampNTZType is absent or shaded differently. If this module is cross-built across multiple Spark versions, consider keeping a version-tolerant check (e.g., match TimestampNTZType when available and fall back to typeName), ideally via a shim utility.

/*

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

🟢 Approval recommended

The changes are cohesive and well-covered by targeted Spark + native regression tests, and the updated validator/signature/type mappings are consistent across JVM and C++ paths.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

Run Gluten Clickhouse CI on x86