Spark: Implement variant extraction pushdown for shredded VARIANT columns

[qlong]

Change

This PR is part of the work to support variant extraction pushdown, the core change is to engineschema that now maps slots to paths in variant.

• Add SparkVariantExtractionScanBuilder implementing SupportsPushDownVariantExtractions so Spark can push variant_get paths from Filter/Project nodes into Iceberg scans.
• Gate behind new spark.sql.iceberg.variant-extraction-push-down.enabled (default on).
• Use an all-or-nothing batch policy: decline the entire batch if any extraction has an unsupported path, unsupported target type, references a non-variant column, or is a full-variant slot (expectedDataType = VariantType, path $).
• Avoid partial scan rewrites that break multi-variant tables and plans where variant_get above join/aggregate barriers still references the original column.
• Override readSchema() on batch query scans to expose annotated extraction structs to executors.
• Add TestVariantShreddingPushdown for DSv2 plan shape and query correctness.

issue: #16448

Notes for reviewers

• PathUtil.java is mostly copied from existing PR Api: Support variant extract and fix manifest bounds byte order #15384, will rebase once that PR is merged.
• Merge order:
  1. Spark: Add selective shredded variant extraction Parquet readers #16714
  2. Spark: Implement variant extraction pushdown for shredded VARIANT columns #16715

End to end testing

Requires #16714 for end-to-end testing. To try the full pushdown + selective read path without merging locally, use this branch:

https://github.com/qlong/iceberg/tree/variant-extraction-integration-test

Test Results

See performance improvements in #16714

Co-authored with Claude Sonnet 4.6

[qlong]

@rdblue@steveloughran@nssalian PTAL when you get a chance.

[qlong]

Spark side fix for column pruning when variant is pushed down. apache/spark#56556

[qlong]

Spark side fix to ensure variant pushdown works through aggregate/join/sort:

apache/spark#57190

[steveloughran]

@qlong now that spark 4.2 has shipped, what's the implications for variants in iceberg?

[qlong]

@steveloughran , I did not see performance related fix for variant in Spark 4.2.

Pending Spark 4.3 has a few performance fix for variant, including variant filter pushdown (pre-requite for your row group skipping on iceberg side), and extraction pushdown enhancement.

In short, file skipping/rowgroup skipping based on filter pushdown won't work until 4.3 is released. Extraction pushdown can work today in simple cases, will automatically get performance enhancement from Spark 4.3.

[qlong]

Spark 4.3 branch was cut, confirmed it contains the extraction pushdown fix (apache/spark#56556) and enhancement (apache/spark#57190), plus the new variant_get in DSv2 expression to support filter pushdown (apache/spark#54394)

[qlong]

ping to prevent auto close

[steveloughran]

we should automate that ping