[SPARK-58062][SQL] Add variant_strip_nulls expression

[bojana-db]

What changes were proposed in this pull request?

Adds the SQL function variant_strip_nulls(v[, includeArrays]), which recursively removes Variant null fields from objects (and, by default, Variant null elements from arrays) in a Variant value.

Details:

• Object path: a field whose value is a Variant null (the JSON null literal) is removed entirely; recursion walks nested objects and arrays at all levels;
• includeArrays (BOOLEAN, default true): when true, Variant null elements are also removed from arrays; when false, array null elements are kept, but null fields of objects nested anywhere (including inside arrays) are still stripped;
• Empty containers are preserved: an object/array emptied by stripping stays {} / []; the parent is never collapsed to NULL;
• Non-container inputs — scalars and a top-level Variant null (parse_json('null')) — are returned unchanged;
• NULL intolerant: any NULL argument returns NULL.

Why are the changes needed?

There is no built-in way to drop Variant null entries from a Variant; today users must convert it to another type (e.g. a map) and back.

Does this PR introduce any user-facing change?

Yes, a new SQL function (and Scala/Python functions API.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code with Claude Opus 4.8

[uros-b]

cc @marcuslin123 regarding possible duplication (#57318 (review)).

[harshmotw-db]

LGTM! I noticed in the tests that all leaf types are small integers (probably always BYTE). Can we add some tests where other data types like strings and wider integers are also mixed in? Thanks

[uros-b]

Why default true? PostgreSQL 18 json_strip_nulls defaults to false (https://www.postgresql.org/docs/devel/functions-json.html):

If strip_in_arrays is true (the default is false), null array elements are also stripped.

[uros-b]

Aslo, note possible issues when stripping array elements ends up renumbering the array. After variant_strip_nulls(v) with the current default, any positional path a user holds ($.a[1]) silently points at different data. That's a surprising side effect for a function that reads like it only removes nulls, so it seems better as opt-in?

[uros-b]

Also, strip_in_arrays sounds a bit better than include_arrays here? cc @srielau@harshmotw-db

[bojana-db]

variant_strip_nulls mirrors BigQuery's JSON_STRIP_NULLS, where include_arrays defaults to TRUE, so name and default both match BigQuery. PostgreSQL's strip_in_arrays=false is a backward-compat default (arrays were untouched before PG 18), not an analytics preference. BigQuery frames the function as data compaction ( "JSON_STRIP_NULLS compresses the data by removing JSON nulls ... helpful for reducing data size during exports" ) which makes sense fot DW/analytics workload and for that use case you generally want array nulls stripped too.

[uros-b]

The value is passed through StaticInvoke and read per row at runtime, so a non-foldable boolean column would work here with no code changes. Could you just please confirm, is this restriction deliberate?

It's also inconsistent with the closest sibling: VariantSet.createIfMissing is a plain QuaternaryExpression child with no foldability check and accepts dynamic expressions, even though its PySpark docstring claims "Must be a constant." So today the family documents the same constraint in two places and enforces it in one.

[bojana-db]

Yes, it is deliberate. It makes sense for a flag to be foldable as it represents the function behaviour mode and in a sense it can be seen as two different functions. Regarding variant_set, restriction is added in #57855.

[uros-b]

LGTM, thank you @bojana-db for resolving all comments. Also, thank you @harshmotw-db for review!

[uros-b]

@bojana-db Please resolve conflicts.

[uros-b]

Merge Summary:

• merged into master 2740493
• merged into branch-4.x 6f99005

Posted by merge_spark_pr.py