Skip to content

[feature](function) Add jaro, jaro_winkler and jaccard_similarity string functions - #67436

Open
puranjay2597 wants to merge 3 commits into
apache:masterfrom
puranjay2597:feature/string-distance-functions
Open

[feature](function) Add jaro, jaro_winkler and jaccard_similarity string functions#67436
puranjay2597 wants to merge 3 commits into
apache:masterfrom
puranjay2597:feature/string-distance-functions

Conversation

@puranjay2597

@puranjay2597puranjay2597 commented Sep 2, 2026

Copy link
Copy Markdown

Supersedes #60799 (GitHub won't allow reopening a PR whose branch was force-pushed after it was closed).

What problem does this PR solve?

Adds 3 built-in scalar functions for fuzzy string matching and similarity scoring, useful for record deduplication, search ranking, and data quality workflows:

FunctionReturn typeDescription
jaro(str1, str2)DOUBLEJaro similarity [0.0, 1.0]
jaro_winkler(str1, str2)DOUBLEJaro-Winkler similarity [0.0, 1.0] (boosts strings sharing a common prefix)
jaccard_similarity(str1, str2)DOUBLEJaccard similarity [0.0, 1.0] over the sets of distinct characters of the two strings

All functions accept VARCHAR/STRING inputs, propagate NULL, and support constant folding.

Rebased and reworked since #60799

#60799 originally also proposed levenshtein and damerau_levenshtein. Both now already exist on master under different names (levenshtein via #60412, damerau_levenshtein_distance via #65278), so they've been dropped from this PR to avoid duplicating functionality — only the 3 functions above remain.

The remaining functions have been reworked from the original submission to address review feedback on #60799:

  • UTF-8 support (requested by @linrrzqqq): all 3 functions now correctly handle multi-byte UTF-8 input instead of operating byte-by-byte, e.g. jaro_winkler('你好世界', '你好世间') now compares by character. Implemented with an ASCII fast path plus a UTF-8-aware path, following the exact pattern established by levenshtein/damerau_levenshtein_distance (VStringFunctions::get_utf8_char_offsets / utf8_char_equal).
  • Reduced duplication (requested by @linrrzqqq): added a standalone jaro function containing the core Jaro algorithm; jaro_winkler now calls it directly instead of duplicating the matching/transposition logic.
  • Jaccard algorithm clarified (requested by @linrrzqqq, who couldn't find the byte-bigram approach in Wikipedia or ClickHouse): jaccard_similarity is now a character-set Jaccard index — |A ∩ B| / |A ∪ B| over the sets of distinct bytes (ASCII) or Unicode characters (UTF-8) — matching ClickHouse's stringJaccardIndex (FunctionsStringDistance.cpp) instead of an unexplained bigram scheme. Uses a std::bitset<256> for the ASCII path per the reviewer's suggestion, and a hash set of UTF-8 characters otherwise.
  • FE constant folding: added jaro/jaro_winkler/jaccard_similarity fold-constant implementations in StringArithmetic.java, matching the convention used by levenshtein/damerau_levenshtein_distance/hamming_distance.
  • Input length guard: all 3 functions reject inputs over 65535 bytes with a clear error, guarding the O(m×n) Jaro matching window and the set construction in jaccard_similarity against unbounded STRING inputs.
  • Expanded tests: BE unit tests (function_string_test.cpp) and regression tests now cover column-vs-column, column-vs-constant (both directions), nullable columns, UTF-8, and over-length-input error cases, in addition to the constant-only cases.

Release note

Add 3 built-in string similarity functions: jaro, jaro_winkler, jaccard_similarity.

Check List (For Author)

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@puranjay2597

Copy link
Copy Markdown
Author

Answering the checklist above:

  1. Problem / what this adds: three new fuzzy string-matching scalar functions — jaro, jaro_winkler, jaccard_similarity. This continues [feat](func) Add levenshtein, damerau_levenshtein, jaro_winkler, jaccard_similarity built-in scalar functions #60799, whose levenshtein/damerau_levenshtein have since landed separately via Add levenshtein and hamming_distance functions #60412 and [Feature](function) Support function DAMERAU_LEVENSHTEIN_DISTANCE #65278, so those two are dropped here to avoid duplicating them.
  2. Behavior changed: none — purely additive, no existing function or code path is touched.
  3. Features added: see (1). jaro_winkler reuses the jaro implementation rather than duplicating it, and jaccard_similarity is a character-set Jaccard index (ASCII bitset / UTF-8 hash set) matching ClickHouse's stringJaccardIndex. All three have UTF-8 support, FE constant folding, and an input-length guard.
    4/5. Refactor / optimization: not applicable, no existing code was refactored.

@puranjay2597
puranjay2597force-pushed the feature/string-distance-functions branch from 2438cce to d7c7947CompareSeptember 2, 2026 07:40
…ing functions
Rebuilds the string-similarity functions proposed in apache#60799 on top of current
master, addressing prior review feedback:
- levenshtein and damerau_levenshtein are dropped: both now exist on master
(apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping
ours would only collide.
- All three functions get full UTF-8 support (ASCII fast path + character-
aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal),
matching the pattern established by levenshtein/damerau_levenshtein_distance
instead of operating on raw bytes.
- jaro_winkler now shares its Jaro computation with the new jaro function
instead of duplicating the matching/transposition logic.
- jaccard_similarity is redefined as a character-set Jaccard index (bitset for
the ASCII path, hash set of UTF-8 characters otherwise), matching
ClickHouse's stringJaccardIndex semantics, rather than an unexplained
byte-bigram scheme.
- Added FE constant-folding (StringArithmetic.java) for all three functions,
and BE unit tests plus expanded regression coverage (column/constant
combinations, nullable columns, UTF-8, over-length-input errors).
@puranjay2597
puranjay2597force-pushed the feature/string-distance-functions branch from d7c7947 to 3f0a865CompareSeptember 2, 2026 08:28
Puranjay Patil added 2 commits September 3, 2026 18:14
…rd with BE
Reviewed the merged levenshtein/hamming_distance (apache#60412) and
damerau_levenshtein_distance (apache#65278, apache#66236) PRs for consistency with this
one. Found and fixed a real FE/BE divergence in the process:
- The FE constant-fold length guard checked Java code point count, while the
BE guard checks UTF-8 byte length. For multi-byte input the two disagree
(e.g. ~30000 3-byte characters is under the FE's 65535 code-point cap but
over BE's 65535-byte cap), so a literal expression could fold successfully
on FE while the same value would be rejected by BE if read from a column.
FE now measures UTF-8 bytes too, matching function_string_similarity.cpp
exactly (mirrors the fix pattern in apache#64881, "Align constant folding with
BE results").
- Added regression coverage for astral-plane (surrogate-pair/4-byte UTF-8)
characters on all three functions, matching the precedent set for
RIGHT/INSTR in apache#64881.
Re-checked function_string_similarity.cpp and the new test cases against
.clang-format's ColumnLimit (100) -- a handful of lines (mostly lambda/call
wrapping and a UTF-8 literal in a test) were over. No logic changes.
@linrrzqqqlinrrzqqq self-assigned this Sep 4, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@puranjay2597@hello-stephen@linrrzqqq