[SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API by HyukjinKwon · Pull Request #57184 · apache/spark

HyukjinKwon · 2026-07-10T07:48:45Z

What changes were proposed in this pull request?

This documents, for every built-in function in the Scala functions API (functions.scala), the
accepted input type of each argument (@param) and the return type of each DataFrame function
(@return), using the numeric umbrella where numeric subtypes cast in and timestamp for
TIMESTAMP_NTZ in type phrases.

Subtask of SPARK-57999, split out from #57079 for reviewability. This PR covers only
the Scala functions.scala surface.

Why are the changes needed?

The Scala functions Scaladoc previously had no per-argument type information and no documented
return type for most functions. This fills that gap.

Does this PR introduce any user-facing change?

Yes, documentation only. No behavioral or API change.

How was this patch tested?

Scala style/format (scalastyle + scalafmt) and the docs build pass. Types were derived from and
verified against the analyzer's behavior and each expression's inputTypes / dataType.

Co-authored-by: Isaac

dongjoon-hyun

Could you make the CI happy?

The scalafmt check failed on sql/connect or sql/connect at following occurrences:
org.apache.maven.plugin.MojoExecutionException: Scalafmt: Unformatted files found
Error: Failed to execute goal org.antipathy:mvn-scalafmt_2.13:1.1.1713302731.c3d0074:format (default-cli) on project spark-sql-api_2.13: Error formatting Scala files: Scalafmt: Unformatted files found -> [Help 1]
Before submitting your change, please make sure to format your code using the following command:
./build/mvn scalafmt:format -Dscalafmt.skip=false -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl sql/api -pl sql/connect/common -pl sql/connect/server -pl sql/connect/shims -pl sql/connect/client/jvm

MaxGekk

0 blocking, 3 non-blocking, 0 nits.
A careful, high-quality documentation pass — the type conventions hold up across the functions I sampled. One @return overstates its type; two smaller doc gaps.

Correctness (1)

functions.scala:13749: from_json non-StructType overloads are documented as returning a struct, but they also accept array/map schemas (verified: they return ArrayType/MapType) — see inline

Suggestions (1)

functions.scala:13628: posexplode/posexplode_outer have a @param but no @return, unlike explode/explode_outer — see inline

Nits (1)

functions.scala:8619: comma splice + "of string" → "of strings" in the search@param (pre-existing, but on a line this PR edits) — see inline

Verification

Confirmed finding #1 by running from_json against a local build: from_json(col, ArrayType(IntegerType)) → ArrayType(IntegerType,true), from_json(col, MapType(StringType,IntegerType)) → MapType(...), and from_json(col, lit("array<int>")) (Column overload) → ArrayType(...) — none a struct — while the StructType overload correctly returns a struct. Spot-checked seven other @returns (size→integer, array_contains→boolean, ceil→long, concat_ws→string, sqrt→double, abs→same-as-input, map_keys→array); all matched the docs.

cloud-fan

0 blocking, 0 non-blocking, 0 nits.
No blocking, non-blocking, or nit findings on the current head.

Verification

Mechanically examined 5,267 added documentation lines for objective prose defects and 2,302 structured type-contract candidates across 1,706 claim blocks. The selected scanners reported complete coverage with no findings; the previously reported documentation issues are corrected on the current head.

…the Scala functions API ### What changes were proposed in this pull request? This documents, for every built-in function in the Scala `functions` API (`functions.scala`), the accepted input type of each argument (`@param`) and the return type of each DataFrame function (`@return`), using the numeric umbrella where numeric subtypes cast in and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only the Scala `functions.scala` surface. ### Why are the changes needed? The Scala `functions` Scaladoc previously had no per-argument type information and no documented return type for most functions. This fills that gap. ### Does this PR introduce _any_ user-facing change? Yes, documentation only. No behavioral or API change. ### How was this patch tested? Scala style/format (scalastyle + scalafmt) and the docs build pass. Types were derived from and verified against the analyzer's behavior and each expression's `inputTypes` / `dataType`. Co-authored-by: Isaac

…@return

…@return, replace grammar, nanvl/kll/to_char types, and scalafmt Co-authored-by: Isaac

Match the SQL and PySpark function reference by noting that the variadic concat_ws argument accepts a string or an array of strings. Co-authored-by: Isaac

Audit surfaced many functions where some arguments (or the return) carried an accepted-type phrase but sibling value arguments did not. Document the missing arguments to match the per-argument coverage promised by the PR, using sibling overloads and the analogous functions as the source of the accepted types. Co-authored-by: Isaac

Cover the accepted types for arguments that were still missing a type phrase in the current_time, radians, hll_sketch_estimate, hll_union, theta_union, and tuple_sketch_agg_* functions, matching their sibling overloads. Co-authored-by: Isaac

…APIs Complete per-argument type coverage for the from_json/to_json/schema_of_json, variant_*, from_csv/to_csv/schema_of_csv, from_xml/to_xml/schema_of_xml, and mask function families, whose returns were documented but whose arguments still lacked accepted-type phrases. Column arguments are documented as evaluating to the accepted type; literal schema/options/path/target-type arguments are noted as constants. Co-authored-by: Isaac

Reflow the added accepted-type doc phrases with scalafmt and add missing terminal punctuation so the merged scaladoc reads correctly. Formatting only. Co-authored-by: Isaac

…he Scala functions API This documents, for every built-in function in the Scala `functions` API (`functions.scala`), the accepted input type of each argument (`param`) and the return type of each DataFrame function (`return`), using the numeric umbrella where numeric subtypes cast in and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from #57079 for reviewability. This PR covers only the Scala `functions.scala` surface. The Scala `functions` Scaladoc previously had no per-argument type information and no documented return type for most functions. This fills that gap. Yes, documentation only. No behavioral or API change. Scala style/format (scalastyle + scalafmt) and the docs build pass. Types were derived from and verified against the analyzer's behavior and each expression's `inputTypes` / `dataType`. Co-authored-by: Isaac Closes#57184 from HyukjinKwon/split-func-types-scala. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 99025ce) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>

HyukjinKwon · 2026-07-21T06:36:48Z

Merge Summary:

merged into master 99025ce
merged into branch-4.x 98e98b7

Posted by merge_spark_pr.py

HyukjinKwon mentioned this pull request Jul 10, 2026
[SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions #57079
Closed

dongjoon-hyun reviewed Jul 10, 2026
View reviewed changes

HyukjinKwon marked this pull request as draft July 12, 2026 22:08

HyukjinKwon force-pushed the split-func-types-scala branch 2 times, most recently from f9b4f14 to 88a7746Compare July 13, 2026 00:01

HyukjinKwon marked this pull request as ready for review July 13, 2026 06:26

MaxGekk reviewed Jul 14, 2026
View reviewed changes

Comment threadsql/api/src/main/scala/org/apache/spark/sql/functions.scala Outdated
Comment threadsql/api/src/main/scala/org/apache/spark/sql/functions.scala
Comment threadsql/api/src/main/scala/org/apache/spark/sql/functions.scala Outdated

This was referenced Jul 19, 2026
[SPARK-58062][SQL] Add variant_strip_nulls expression #57318
Closed
[SPARK-58186][SQL] Add bitmap_contains function #57336
Closed

cloud-fan approved these changes Jul 20, 2026
View reviewed changes

HyukjinKwon added 3 commits July 21, 2026 07:42

Address review comments: from_json struct/array/map return, posexplode…
4147004
…@return, replace grammar, nanvl/kll/to_char types, and scalafmt Co-authored-by: Isaac

Document concat_ws array-of-strings argument for cross-API consistency
fb39c06
Match the SQL and PySpark function reference by noting that the variadic concat_ws argument accepts a string or an array of strings. Co-authored-by: Isaac

HyukjinKwon force-pushed the split-func-types-scala branch from 7b57f59 to fb39c06Compare July 20, 2026 22:49

HyukjinKwon added 4 commits July 21, 2026 11:12

Apply scalafmt to functions API doc changes
ebf56de
Reflow the added accepted-type doc phrases with scalafmt and add missing terminal punctuation so the merged scaladoc reads correctly. Formatting only. Co-authored-by: Isaac

HyukjinKwon closed this in 99025ceJul 21, 2026

Navigation Menu

Uh oh!

[SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API - #57184

[SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API#57184
HyukjinKwon wants to merge 7 commits into
apache:masterfrom
HyukjinKwon:split-func-types-scala

HyukjinKwon commented Jul 10, 2026

Uh oh!

dongjoon-hyun left a comment

Uh oh!

MaxGekk left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Uh oh!

HyukjinKwon commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

HyukjinKwon commented Jul 10, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk left a comment

Choose a reason for hiding this comment

Correctness (1)

Suggestions (1)

Nits (1)

Verification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Verification

Uh oh!

HyukjinKwon commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants