Uh oh!
There was an error while loading. Please reload this page.
[feature](function) Add jaro, jaro_winkler and jaccard_similarity string functions - #67436
Open
puranjay2597 wants to merge 3 commits into
Open
[feature](function) Add jaro, jaro_winkler and jaccard_similarity string functions#67436puranjay2597 wants to merge 3 commits into
puranjay2597 wants to merge 3 commits into
Conversation
puranjay2597
requested review from
924060929, englefly, morrySnow and starocean999
as code ownersSeptember 2, 2026 07:22
hello-stephen
commented
Sep 2, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
9 tasks
puranjay2597force-pushed
the
feature/string-distance-functions
branch
from
September 2, 2026 07:26
c2cb8c0 to
2438cceComparepuranjay2597
commented
Sep 2, 2026
Author
Answering the checklist above:
|
puranjay2597force-pushed
the
feature/string-distance-functions
branch
from
September 2, 2026 07:40
2438cce to
d7c7947Compare…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors).
puranjay2597force-pushed
the
feature/string-distance-functions
branch
from
September 2, 2026 08:28
d7c7947 to
3f0a865Compare
added 2 commits
September 3, 2026 18:14
…rd with BE Reviewed the merged levenshtein/hamming_distance (apache#60412) and damerau_levenshtein_distance (apache#65278, apache#66236) PRs for consistency with this one. Found and fixed a real FE/BE divergence in the process: - The FE constant-fold length guard checked Java code point count, while the BE guard checks UTF-8 byte length. For multi-byte input the two disagree (e.g. ~30000 3-byte characters is under the FE's 65535 code-point cap but over BE's 65535-byte cap), so a literal expression could fold successfully on FE while the same value would be rejected by BE if read from a column. FE now measures UTF-8 bytes too, matching function_string_similarity.cpp exactly (mirrors the fix pattern in apache#64881, "Align constant folding with BE results"). - Added regression coverage for astral-plane (surrogate-pair/4-byte UTF-8) characters on all three functions, matching the precedent set for RIGHT/INSTR in apache#64881.
Re-checked function_string_similarity.cpp and the new test cases against .clang-format's ColumnLimit (100) -- a handful of lines (mostly lambda/call wrapping and a UTF-8 literal in a test) were over. No logic changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #60799 (GitHub won't allow reopening a PR whose branch was force-pushed after it was closed).
What problem does this PR solve?
Adds 3 built-in scalar functions for fuzzy string matching and similarity scoring, useful for record deduplication, search ranking, and data quality workflows:
jaro(str1, str2)[0.0, 1.0]jaro_winkler(str1, str2)[0.0, 1.0](boosts strings sharing a common prefix)jaccard_similarity(str1, str2)[0.0, 1.0]over the sets of distinct characters of the two stringsAll functions accept
VARCHAR/STRINGinputs, propagate NULL, and support constant folding.Rebased and reworked since #60799
#60799 originally also proposed
levenshteinanddamerau_levenshtein. Both now already exist on master under different names (levenshteinvia #60412,damerau_levenshtein_distancevia #65278), so they've been dropped from this PR to avoid duplicating functionality — only the 3 functions above remain.The remaining functions have been reworked from the original submission to address review feedback on #60799:
jaro_winkler('你好世界', '你好世间')now compares by character. Implemented with an ASCII fast path plus a UTF-8-aware path, following the exact pattern established bylevenshtein/damerau_levenshtein_distance(VStringFunctions::get_utf8_char_offsets/utf8_char_equal).jarofunction containing the core Jaro algorithm;jaro_winklernow calls it directly instead of duplicating the matching/transposition logic.jaccard_similarityis now a character-set Jaccard index —|A ∩ B| / |A ∪ B|over the sets of distinct bytes (ASCII) or Unicode characters (UTF-8) — matching ClickHouse'sstringJaccardIndex(FunctionsStringDistance.cpp) instead of an unexplained bigram scheme. Uses astd::bitset<256>for the ASCII path per the reviewer's suggestion, and a hash set of UTF-8 characters otherwise.jaro/jaro_winkler/jaccard_similarityfold-constant implementations inStringArithmetic.java, matching the convention used bylevenshtein/damerau_levenshtein_distance/hamming_distance.jaccard_similarityagainst unboundedSTRINGinputs.function_string_test.cpp) and regression tests now cover column-vs-column, column-vs-constant (both directions), nullable columns, UTF-8, and over-length-input error cases, in addition to the constant-only cases.Release note
Add 3 built-in string similarity functions:
jaro,jaro_winkler,jaccard_similarity.Check List (For Author)
Test
Behavior changed:
Does this need documentation?