[SPARK-58087][PYTHON][DOCS] Document accepted input and return types in the PySpark functions API by HyukjinKwon · Pull Request #57185 · apache/spark

HyukjinKwon · 2026-07-10T07:48:56Z

What changes were proposed in this pull request?

This documents, for every built-in function in the PySpark functions API (builtin.py), the
accepted input type of each argument (Parameters) and the return type of each function
(Returns), using the numeric umbrella where numeric subtypes cast in and timestamp for
TIMESTAMP_NTZ in type phrases.

Subtask of SPARK-57999, split out from #57079 for reviewability. This PR covers only
the PySpark builtin.py surface.

Why are the changes needed?

The PySpark functions docstrings previously had no per-argument type information for most
functions. This fills that gap.

Does this PR introduce any user-facing change?

Yes, documentation only. No behavioral or API change.

How was this patch tested?

Python lint (flake8 + black) and the docstring doctests pass. Types were derived from and verified
against the analyzer's behavior and each expression's inputTypes / dataType.

Co-authored-by: Isaac

HyukjinKwon · 2026-07-10T23:56:08Z

Thank you @dongjoon-hyun for review. Let me try to take a look throughoutly.

dongjoon-hyun

For kll_sketch_get_quantile_bigint/_float/_double, kll_sketch_get_rank_*, the return type is sometimes array.

cloud-fan

1 blocking, 0 non-blocking, 0 nits.
The documentation sweep is broadly consistent, but one family of timestamp constructors still has undocumented argument types.

Correctness (1)

python/pyspark/sql/functions/builtin.py:26638: Document the remaining timezone/date/time argument types in the timestamp constructor family -- see inline

Verification

Compared the timestamp constructor docstrings with MakeTimestamp.inputTypes and MakeTimestampFromDateTime.inputTypes in Catalyst. These require string timezone, date, and time inputs for the parameters whose type phrases are still absent.

…in the PySpark functions API ### What changes were proposed in this pull request? This documents, for every built-in function in the PySpark `functions` API (`builtin.py`), the accepted input type of each argument (`Parameters`) and the return type of each function (`Returns`), using the numeric umbrella where numeric subtypes cast in and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only the PySpark `builtin.py` surface. ### Why are the changes needed? The PySpark `functions` docstrings previously had no per-argument type information for most functions. This fills that gap. ### Does this PR introduce _any_ user-facing change? Yes, documentation only. No behavioral or API change. ### How was this patch tested? Python lint (flake8 + black) and the docstring doctests pass. Types were derived from and verified against the analyzer's behavior and each expression's `inputTypes` / `dataType`. Co-authored-by: Isaac

… to_char constant note, and broaden date extractor input types Co-authored-by: Isaac

…uctors Add the accepted-type phrases for the timezone (string), date, and time arguments across the make_timestamp / try_make_timestamp family, matching the Catalyst MakeTimestamp and MakeTimestampFromDateTime inputTypes contracts. Co-authored-by: Isaac

Match the SQL and Scala function reference by noting that the variadic concat_ws argument accepts a string or an array of strings. Co-authored-by: Isaac

Audit surfaced many functions where some arguments (or the return) carried an accepted-type phrase but sibling value arguments did not. Document the missing arguments and returns to match the per-argument coverage promised by the PR, keeping the accepted types consistent with the SQL and Scala function reference. Co-authored-by: Isaac

Add the accepted-type phrase to the Returns section of stddev and its std alias, matching the sibling aggregate functions. Co-authored-by: Isaac

Bring the from_json/to_json/schema_of_json, variant_*, from_csv/to_csv/ schema_of_csv, and from_xml/to_xml/schema_of_xml function families into the accepted-type documentation convention, documenting both their arguments and their return values to match the Scala functions API. Co-authored-by: Isaac

…n the PySpark functions API ### What changes were proposed in this pull request? This documents, for every built-in function in the PySpark `functions` API (`builtin.py`), the accepted input type of each argument (`Parameters`) and the return type of each function (`Returns`), using the numeric umbrella where numeric subtypes cast in and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from #57079 for reviewability. This PR covers only the PySpark `builtin.py` surface. ### Why are the changes needed? The PySpark `functions` docstrings previously had no per-argument type information for most functions. This fills that gap. ### Does this PR introduce _any_ user-facing change? Yes, documentation only. No behavioral or API change. ### How was this patch tested? Python lint (flake8 + black) and the docstring doctests pass. Types were derived from and verified against the analyzer's behavior and each expression's `inputTypes` / `dataType`. Co-authored-by: Isaac Closes#57185 from HyukjinKwon/split-func-types-python. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit ddd66fa) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>

HyukjinKwon · 2026-07-21T06:37:41Z

Merge Summary:

merged into master ddd66fa
merged into branch-4.x 270a501

Posted by merge_spark_pr.py

HyukjinKwon mentioned this pull request Jul 10, 2026
[SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions #57079
Closed

dongjoon-hyun requested changes Jul 10, 2026
View reviewed changes

HyukjinKwon marked this pull request as draft July 12, 2026 22:08

HyukjinKwon force-pushed the split-func-types-python branch 2 times, most recently from 241f910 to d9b102bCompare July 13, 2026 00:01

HyukjinKwon marked this pull request as ready for review July 13, 2026 06:26

dongjoon-hyun reviewed Jul 14, 2026
View reviewed changes

Comment threadpython/pyspark/sql/functions/builtin.py Outdated

dongjoon-hyun reviewed Jul 14, 2026
View reviewed changes

Comment threadpython/pyspark/sql/functions/builtin.py

dongjoon-hyun reviewed Jul 14, 2026
View reviewed changes

Comment threadpython/pyspark/sql/functions/builtin.py Outdated

This was referenced Jul 19, 2026
[SPARK-58062][SQL] Add variant_strip_nulls expression #57318
Closed
[SPARK-58186][SQL] Add bitmap_contains function #57336
Closed
[MINOR][PYTHON][DOCS] Fixed some links in documentation. #57359
Closed

cloud-fan reviewed Jul 20, 2026
View reviewed changes

Comment threadpython/pyspark/sql/functions/builtin.py

HyukjinKwon added 4 commits July 21, 2026 07:42

Address review comments: fix kll array returns, make_interval fields,…
1e556ff
… to_char constant note, and broaden date extractor input types Co-authored-by: Isaac

Document concat_ws variadic argument type for cross-API consistency
1c55308
Match the SQL and Scala function reference by noting that the variadic concat_ws argument accepts a string or an array of strings. Co-authored-by: Isaac

HyukjinKwon force-pushed the split-func-types-python branch from 423fd0b to 1c55308Compare July 20, 2026 22:49

HyukjinKwon added 3 commits July 21, 2026 11:25

Address review: document stddev/std return type
8288056
Add the accepted-type phrase to the Returns section of stddev and its std alias, matching the sibling aggregate functions. Co-authored-by: Isaac

zhengruifeng approved these changes Jul 21, 2026
View reviewed changes

HyukjinKwon closed this in ddd66faJul 21, 2026

art000109 mentioned this pull request Jul 25, 2026
[SPARK-XXXXX][PYTHON][DOCS] Added Returns and See Also sections in pyspark.sql.functions documentation. #57529
Open

Navigation Menu

Uh oh!

[SPARK-58087][PYTHON][DOCS] Document accepted input and return types in the PySpark functions API - #57185

[SPARK-58087][PYTHON][DOCS] Document accepted input and return types in the PySpark functions API#57185
HyukjinKwon wants to merge 7 commits into
apache:masterfrom
HyukjinKwon:split-func-types-python

HyukjinKwon commented Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Jul 10, 2026

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Uh oh!

Uh oh!

HyukjinKwon commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

HyukjinKwon commented Jul 10, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Jul 10, 2026

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Correctness (1)

Verification

Uh oh!

Uh oh!

HyukjinKwon commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants