[SPARK-58275][SQL][PYTHON] Add `normalize` Unicode normalization SQL function by SreeramaYeshwanthGowd · Pull Request #57450 · apache/spark

SreeramaYeshwanthGowd · 2026-07-23T01:08:12Z

What changes were proposed in this pull request?

Add a built in normalize(str[, form]) scalar function that performs Unicode normalization, backed by Spark's bundled ICU4J library.

API surface added:

SQL: normalize(str) and normalize(str, form)
Scala DataFrame: functions.normalize(col) and functions.normalize(col, form)
PySpark, classic and Spark Connect: pyspark.sql.functions.normalize(str, form=None)

Key design choices:

The default form is NFC. Supported forms are NFC, NFD, NFKC, and NFKD, case insensitive, matching the ANSI SQL standard and PostgreSQL, Trino, and BigQuery. Forms follow Unicode Standard Annex [SPARK-1210] Prevent ContextClassLoader of Actor from becoming ClassLoader of Executo... #15's decomposition/composition algorithm.
The form argument is a regular string expression or column, not required to be foldable, consistent with how other Spark functions take a mode string (for example date_trunc). This keeps the change small and idiomatic and avoids adding new grammar. An invalid form raises a clear INVALID_PARAMETER_VALUE.NORMALIZE_FORM error.
Implemented as a RuntimeReplaceable expression backed by a StaticInvoke into ExpressionImplUtils, so there is no hand written codegen, following the recent hmac function.
Normalization uses ICU4J (the same library backing Spark's collation support) rather than the JDK's java.text.Normalizer, so behavior is pinned to Spark's bundled ICU4J/Unicode data and does not vary across JVM vendors or versions. This is documented on the public SQL/Scala/PySpark doc surfaces.

Why are the changes needed?

Unicode normalization is a common need in text processing and data cleaning, for example to compare strings that look identical but differ in code point composition. Spark has no built in way to do this, so users fall back to a UDF, which cannot be optimized by Catalyst. This function is part of the SQL standard and is already available in PostgreSQL, Trino, and Google BigQuery, so it also improves parity with the engines that Spark users migrate from.

Does this PR introduce any user-facing change?

Yes. It adds a new built in SQL function normalize and the corresponding Scala and PySpark DataFrame API entries. No existing behavior changes.

Example:

spark-sql> SELECT normalize('ﬁ', 'NFKC');
fi

How was this patch tested?

Added unit tests in ExpressionImplUtilsSuite and StringExpressionsSuite covering all four forms (NFC, NFD, NFKC, NFKD), a case insensitive form name, null propagation, empty input, a supplementary (surrogate pair) character, a canonical singleton decomposition, a compatibility decomposition, the invalid form error, and a conformance vector for canonical reordering of combining marks with different combining classes (Unicode Standard Annex #15). Added a DataFrame API test in StringFunctionsSuite, a PySpark test in test_functions.py, and SQL golden tests in string-functions.sql with regenerated results. Also regenerated sql-expression-schema.md.

Was this patch authored or co-authored using generative AI tooling? No

uros-b

Thank you @SreeramaYeshwanthGowd! Please fix the cut-off PR title, otherwise LGTM

SreeramaYeshwanthGowd · 2026-07-23T10:50:45Z

Thanks @uros-b! Fixed the PR title.

…function

HyukjinKwon · 2026-08-06T22:32:35Z

+ > SELECT _FUNC_('ﬁ', 'NFKC');
+ fi
+ """,
+ since = "4.3.0",


4.4.0? since branch-4.3 is cut out

…nch-4.3 is cut

SreeramaYeshwanthGowd · 2026-08-11T03:51:32Z

@cloud-fan Would you have a moment to review this when you get a chance? Thank you!

cloud-fan

1 blocking, 0 non-blocking, 0 nits.
The API wiring is coherent, but the public normalization semantics and Unicode-version policy must be defined before merge.

Correctness (1)

Blocking: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:979: Define the normalization algorithm and Unicode-version policy as Spark semantics, then make the JDK-backed implementation and conformance tests satisfy that contract. -- see inline

Verification

Traced SQL registration and every public wrapper to the single Catalyst Normalize expression, then followed its StaticInvoke into java.text.Normalizer. The source defines accepted form names, defaults, null behavior, and invalid-form errors, but neither the public contract nor implementation establishes a Spark-owned Unicode normalization/version policy. No tests were run as part of this review.

PR metadata suggestions

Correct the design-choice bullet that calls form foldable: the implementation and public DataFrame signatures accept any string expression or column, and do not enforce foldability.

cloud-fan · 2026-08-11T09:21:53Z

+@ExpressionDescription(
+ usage = """
+ _FUNC_(str[, form]) - Returns the Unicode normalization of `str` using the normalization `form`.
+ Valid forms are 'NFC' (default), 'NFD', 'NFKC', and 'NFKD'. The form name is case-insensitive.


Blocking:
Please define normalization as a Spark semantic instead of inheriting whichever Unicode tables the current JDK ships. Spell out the canonical/compatibility decomposition and composition behavior for each form, and pin or otherwise document the Unicode-version policy with a version-sensitive conformance test. Otherwise the same query can return different text across supported JVM runtimes as their Unicode data evolves.

…ode version policy Address review feedback: normalize() now uses ICU4J's Normalizer2 (the same library backing collation) instead of java.text.Normalizer, so results no longer vary with the JVM's bundled Unicode data. Documents the Unicode Standard Annex apache#15 semantics and this pinning policy on the SQL/Scala/PySpark docs, and adds a conformance test for canonical combining-mark reordering.

SreeramaYeshwanthGowd · 2026-08-11T11:07:41Z

@cloud-fan Thanks, good catch. normalize() now uses ICU4J (same as collation) instead of java.text.Normalizer, so it's pinned to Spark's bundled Unicode data rather than the JVM's. Documented that and added a conformance test for combining-mark reordering. Also fixed the stale wording in the description.

cloud-fan

1 addressed, 0 remaining, 0 new to this AI review.
0 blocking, 0 non-blocking, 0 nits.
The prior Unicode-version-policy concern is addressed, and the current API, implementation, documentation, and test coverage are coherent.

Verification

Traced SQL registration and every public wrapper to the single Catalyst Normalize expression, then followed its StaticInvoke into the ICU4J-backed helper. Verified that the expression owns the NFC default and string typing, the helper owns all four form mappings and the structured invalid-form error, and the tests exercise the documented Unicode semantics and public entry paths. No tests were run as part of this review.

SreeramaYeshwanthGowd · 2026-08-11T14:03:24Z

@cloud-fan CI is green. Thans for the review.

…function ### What changes were proposed in this pull request? Add a built in `normalize(str[, form])` scalar function that performs Unicode normalization, backed by Spark's bundled ICU4J library. API surface added: - SQL: `normalize(str)` and `normalize(str, form)` - Scala DataFrame: `functions.normalize(col)` and `functions.normalize(col, form)` - PySpark, classic and Spark Connect: `pyspark.sql.functions.normalize(str, form=None)` Key design choices: - The default form is NFC. Supported forms are NFC, NFD, NFKC, and NFKD, case insensitive, matching the ANSI SQL standard and PostgreSQL, Trino, and BigQuery. Forms follow Unicode Standard Annex #15's decomposition/composition algorithm. - The `form` argument is a regular string expression or column, not required to be foldable, consistent with how other Spark functions take a mode string (for example `date_trunc`). This keeps the change small and idiomatic and avoids adding new grammar. An invalid form raises a clear `INVALID_PARAMETER_VALUE.NORMALIZE_FORM` error. - Implemented as a `RuntimeReplaceable` expression backed by a `StaticInvoke` into `ExpressionImplUtils`, so there is no hand written codegen, following the recent `hmac` function. - Normalization uses ICU4J (the same library backing Spark's collation support) rather than the JDK's `java.text.Normalizer`, so behavior is pinned to Spark's bundled ICU4J/Unicode data and does not vary across JVM vendors or versions. This is documented on the public SQL/Scala/PySpark doc surfaces. ### Why are the changes needed? Unicode normalization is a common need in text processing and data cleaning, for example to compare strings that look identical but differ in code point composition. Spark has no built in way to do this, so users fall back to a UDF, which cannot be optimized by Catalyst. This function is part of the SQL standard and is already available in PostgreSQL, Trino, and Google BigQuery, so it also improves parity with the engines that Spark users migrate from. ### Does this PR introduce _any_ user-facing change? Yes. It adds a new built in SQL function `normalize` and the corresponding Scala and PySpark DataFrame API entries. No existing behavior changes. Example: ``` spark-sql> SELECT normalize('ﬁ', 'NFKC'); fi ``` ### How was this patch tested? Added unit tests in `ExpressionImplUtilsSuite` and `StringExpressionsSuite` covering all four forms (NFC, NFD, NFKC, NFKD), a case insensitive form name, null propagation, empty input, a supplementary (surrogate pair) character, a canonical singleton decomposition, a compatibility decomposition, the invalid form error, and a conformance vector for canonical reordering of combining marks with different combining classes (Unicode Standard Annex #15). Added a DataFrame API test in `StringFunctionsSuite`, a PySpark test in `test_functions.py`, and SQL golden tests in `string-functions.sql` with regenerated results. Also regenerated `sql-expression-schema.md`. ### Was this patch authored or co-authored using generative AI tooling? No Closes#57450 from SreeramaYeshwanthGowd/add-normalize-function. Authored-by: SreeramaYeshwanthGowd <yeshwanthgowdsreerama@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 2572d80) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan · 2026-08-11T14:41:04Z

Merge Summary:

merged into master 2572d80
merged into branch-4.x 48af256

Posted by merge_spark_pr.py

SreeramaYeshwanthGowd force-pushed the add-normalize-function branch from 6697570 to 285b6bfCompare July 23, 2026 03:04

SreeramaYeshwanthGowd changed the title ~~[SPARK-58275][SQL][PYTHON] Add normalize Unicode normalization SQL …~~[SPARK-58275][SQL][PYTHON] Add normalize Unicode normalization SQL …Jul 23, 2026

uros-b approved these changes Jul 23, 2026
View reviewed changes

SreeramaYeshwanthGowd changed the title ~~[SPARK-58275][SQL][PYTHON] Add normalize Unicode normalization SQL …~~[SPARK-58275][SQL][PYTHON] Add normalize Unicode normalization SQL functionJul 23, 2026

[SPARK-58275][SQL][PYTHON] Add normalize Unicode normalization SQL …
221d462
…function

SreeramaYeshwanthGowd force-pushed the add-normalize-function branch from 285b6bf to 221d462Compare August 6, 2026 15:40

HyukjinKwon reviewed Aug 6, 2026
View reviewed changes

[SPARK-58275][SQL][PYTHON] Update since version to 4.4.0 now that bra…
8cc9869
…nch-4.3 is cut

SreeramaYeshwanthGowd requested a review from HyukjinKwon August 6, 2026 23:06

Retrigger CI
c124664

cloud-fan reviewed Aug 11, 2026
View reviewed changes

SreeramaYeshwanthGowd requested a review from cloud-fan August 11, 2026 11:09

cloud-fan approved these changes Aug 11, 2026
View reviewed changes

cloud-fan closed this in 2572d80Aug 11, 2026

Navigation Menu

Uh oh!

[SPARK-58275][SQL][PYTHON] Add `normalize` Unicode normalization SQL function - #57450

[SPARK-58275][SQL][PYTHON] Add `normalize` Unicode normalization SQL function#57450
SreeramaYeshwanthGowd wants to merge 4 commits into
apache:masterfrom
SreeramaYeshwanthGowd:add-normalize-function

SreeramaYeshwanthGowd commented Jul 23, 2026•
edited
Loading

Uh oh!

uros-b left a comment

Uh oh!

SreeramaYeshwanthGowd commented Jul 23, 2026

Uh oh!

HyukjinKwonAug 6, 2026

Uh oh!

SreeramaYeshwanthGowd commented Aug 11, 2026

Uh oh!

cloud-fan left a comment

Uh oh!

cloud-fanAug 11, 2026

Uh oh!

SreeramaYeshwanthGowd commented Aug 11, 2026•
edited
Loading

Uh oh!

cloud-fan left a comment

Uh oh!

SreeramaYeshwanthGowd commented Aug 11, 2026

Uh oh!

cloud-fan commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

SreeramaYeshwanthGowd commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling? No

Uh oh!

uros-b left a comment

Choose a reason for hiding this comment

Uh oh!

SreeramaYeshwanthGowd commented Jul 23, 2026

Uh oh!

HyukjinKwonAug 6, 2026

Choose a reason for hiding this comment

Uh oh!

SreeramaYeshwanthGowd commented Aug 11, 2026

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Correctness (1)

Verification

PR metadata suggestions

Uh oh!

cloud-fanAug 11, 2026

Choose a reason for hiding this comment

Uh oh!

SreeramaYeshwanthGowd commented Aug 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Verification

Uh oh!

SreeramaYeshwanthGowd commented Aug 11, 2026

Uh oh!

cloud-fan commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SreeramaYeshwanthGowd commented Jul 23, 2026•
edited
Loading

SreeramaYeshwanthGowd commented Aug 11, 2026•
edited
Loading