Uh oh!
There was an error while loading. Please reload this page.
[feat](func) Add levenshtein, damerau_levenshtein, jaro_winkler, jaccard_similarity built-in scalar functions - #60799
Conversation
Thearas
commented
Feb 24, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
4a5b6aa to
cdf1488Compare
puranjay2597
left a comment
There was a problem hiding this comment.
Added 5 string similarity/distance functions as C++ built-ins: levenshtein, damerau_levenshtein, jaro_winkler, cosine_similarity, jaccard_similarity. Manually verified correctness (test cases in PR body). Clang-format applied. No changes to existing functions.
4d8bd34 to
85803f2Compare
zclllyybb
left a comment
There was a problem hiding this comment.
please add be-ut and regression-test cases like other functions. then I will look at it carefully.
zclllyybb
commented
Feb 24, 2026
btw, cosine_similarity is implemented in #60403. and I'd like to know why you add functions like |
85803f2 to
e64286eComparepuranjay2597
commented
Feb 24, 2026
@zclllyybb - I've added be-ut and regression-test cases, and also removed cosine_similarity since it conflicts with #60403. Regarding damerau_levenshtein — ClickHouse has [damerauLevenshteinDistance] (https://clickhouse.com/docs/en/sql-reference/functions/string-functions#dameraulevensteindistance). The key difference from standard Levenshtein is that a single adjacent transposition (e.g. 'ab' → 'ba') costs 1 instead of 2, which matters for natural-language typo correction and deduplication |
zclllyybb
commented
Feb 25, 2026
run buildall |
There was a problem hiding this comment.
Pull request overview
Adds four new fuzzy string distance/similarity scalar functions to Doris (FE function definitions + BE vectorized implementations), along with regression and unit tests.
Changes:
- Register new built-in scalar functions:
levenshtein,damerau_levenshtein,jaro_winkler,jaccard_similarityin FE (Nereids) and BE. - Implement the functions in BE vectorized string function module.
- Add regression coverage and BE unit tests for core/edge cases (empty strings, NULL propagation, basic known examples).
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/function_p0/test_string_distance_functions.groovy | Adds SQL-based regression cases for the four functions |
| regression-test/data/function_p0/test_string_distance_functions.out | Golden outputs for the new regression suite |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/ScalarFunctionVisitor.java | Adds visitor hooks for the new scalar functions |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Levenshtein.java | FE scalar function definition + signatures |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/DamerauLevenshtein.java | FE scalar function definition + signatures |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/JaroWinkler.java | FE scalar function definition + signatures |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/JaccardSimilarity.java | FE scalar function definition + signatures (incl. Javadoc) |
| fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java | Registers the new functions as built-ins |
| be/src/vec/functions/function_string.cpp | Implements and registers the four BE vectorized string functions |
| be/test/vec/function/function_string_test.cpp | Adds BE unit tests validating expected results and NULL handling |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import java.util.List; | ||
| /** | ||
| * ScalarFunction 'jaccard_similarity'. Returns Jaccard similarity (0.0-1.0) based on character bigrams. |
There was a problem hiding this comment.
The Javadoc says Jaccard similarity is based on “character bigrams”, but the implementation in BE builds bigrams from raw bytes. Please align the documentation with the actual semantics (e.g., “byte bigrams”) to avoid confusion for UTF-8 input.
| * ScalarFunction'jaccard_similarity'. ReturnsJaccardsimilarity (0.0-1.0) basedoncharacterbigrams. | |
| * ScalarFunction'jaccard_similarity'. ReturnsJaccardsimilarity (0.0-1.0) basedonbytebigramsoftheinputstrings. |
| const size_t stride = n + 2; | ||
| const int32_t max_dist = static_cast<int32_t>(m + n); | ||
| std::vector<int32_t> d((m + 2) * stride, 0); | ||
| d[0] = max_dist; |
There was a problem hiding this comment.
DamerauLevenshteinOp allocates a full (m+2)×(n+2) int32_t matrix based directly on input lengths. With VARCHAR potentially up to 65533 bytes, this can attempt multi‑GB allocations (or overflow size computations) and lead to OOM/instability. Please add an explicit guard (max input length and/or max matrix cells with overflow-safe multiplication) and fail gracefully (e.g., throw a controlled exception or return NULL) before allocating.
| static void execute(const std::string_view& s, const std::string_view& t, int32_t& res) { | ||
| const size_t m = s.size(); | ||
| const size_t n = t.size(); |
There was a problem hiding this comment.
LevenshteinOp is O(m×n) time and can become prohibitively expensive on large VARCHAR/STRING inputs (e.g., tens of thousands of bytes). Consider adding a length/cost guard (similar to other expensive functions) to prevent runaway CPU usage or query timeouts when users pass very long strings.
| constsize_t n = t.size(); | |
| constsize_t n = t.size(); | |
| // Guard against excessively expensive O(m * n) computation on very large inputs. | |
| // If the estimated cost exceeds the threshold, return the maximal possible | |
| // Levenshtein distance for these lengths (max(m, n)) without running DP. | |
| constexprsize_tMAX_LEVENSHTEIN_COST = 4 * 1024 * 1024; // tunable upper bound | |
| if (static_cast<unsignedlonglong>(m) * static_cast<unsignedlonglong>(n) > | |
| static_cast<unsignedlonglong>(MAX_LEVENSHTEIN_COST)) { | |
| res = static_cast<int32_t>(std::max(m, n)); | |
| return; | |
| } |
| if (s.size() < 2 && t.size() < 2) { | ||
| res = 0.0; | ||
| return; | ||
| } |
There was a problem hiding this comment.
In JaccardSimilarityOp, the two consecutive checks for size() < 2 both return 0.0 (except for the earlier s == t fast path), so one of them is redundant. Simplifying this branch would make the edge-case behavior easier to follow and slightly reduce branching.
| if (s.size() < 2 && t.size() < 2) { | |
| res = 0.0; | |
| return; | |
| } |
e64286e to
dd9de08Comparehello-stephen
commented
Feb 25, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
puranjay2597
commented
Feb 25, 2026
@zclllyybb - I force-pushed to address FE UT checkstyle failure and Copilot AI review comments |
dd9de08 to
0c3b86aComparezclllyybb
commented
Feb 25, 2026
run buildall |
zclllyybb
commented
Feb 25, 2026
@puranjay2597 got it. I will review this pr in few days |
doris-robot
commented
Feb 25, 2026
TPC-H: Total hot run time: 28788 ms |
doris-robot
commented
Feb 25, 2026
TPC-DS: Total hot run time: 183534 ms |
doris-robot
commented
Feb 25, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Feb 25, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
0c3b86a to
54ddeddComparezclllyybb
commented
Mar 3, 2026
run buildall |
zclllyybb
commented
Mar 3, 2026
/review |
There was a problem hiding this comment.
Code Review Summary
This PR adds 4 new built-in scalar functions for fuzzy string matching: levenshtein, damerau_levenshtein, jaro_winkler, and jaccard_similarity. The implementation follows existing patterns (Op struct + StringFunctionImpl + FunctionBinaryToType template chain) consistently. Algorithm implementations have been verified correct. FE registration follows established conventions.
Critical Checkpoint Conclusions
Goal & Tests: The PR achieves its stated goal. BE unit tests, regression tests, and FE registration are all present. However, the regression tests lack negative/error test cases (e.g., test { sql ...; exception ... } for over-limit inputs).
Concurrency: Not applicable — pure scalar functions with no shared state.
Lifecycle management: Not applicable.
Configuration items: None added.
Incompatible changes: None — purely additive new functions.
Parallel code paths: All 4 functions follow the same pattern. However, jaro_winkler and jaccard_similarity are missing input length guards that levenshtein and damerau_levenshtein correctly implement (see inline comments).
Test coverage: Good basic coverage for happy-path cases. Missing: (1) error cases for over-limit inputs via test { ... exception ... } in regression tests, (2) the MARTHA test case is not in the BE unit test for jaro_winkler (only in regression test).
Performance: The missing length guard on jaro_winkler is the main concern — its O(m*n) matching loop with no cap on STRING-type inputs (up to ~2GB) can cause unbounded CPU usage and OOM from heap allocation. See inline comment for details.
Issues Found
[HIGH]
JaroWinklerOpmissing input length guard — O(m*n) time complexity with no input cap; STRING type allows up to ~2GB inputs. This is inconsistent with the guards inLevenshteinOp(65535) andDamerauLevenshteinOp(10000), and can cause DoS-like resource exhaustion per row.[MEDIUM]
JaccardSimilarityOpmissing input length guard — While its core algorithm is O(m log m), the heap-allocatedstd::vector<uint16_t>(m-1)for very large STRING inputs could cause OOM. Less urgent than Jaro-Winkler since computation is not quadratic, but should be consistent with the other functions.[LOW] Regression tests lack error/negative test cases — Per Doris test standards, expected error cases should use
test { sql "..."; exception "..." }pattern. Thelevenshteinanddamerau_levenshteinfunctions throw on over-limit inputs, but this is not tested.
| res = 0.0; | ||
| return; | ||
| } | ||
| const size_t match_dist = std::max(m, n) / 2 - (std::max(m, n) >= 2 ? 1 : 0); |
There was a problem hiding this comment.
[HIGH] Missing input length guard.JaroWinklerOp has O(m * match_dist) = O(m * n/2) time complexity in its matching loop (lines 453-462 in the source), but unlike LevenshteinOp (capped at 65535 bytes) and DamerauLevenshteinOp (capped at 10000 bytes), there is no length guard here.
Since the function accepts STRING type (up to ~2GB), a malicious or accidental query like SELECT jaro_winkler(repeat('x', 100000), repeat('y', 100000)) would perform ~5 billion iterations per row. Additionally, the heap allocation at the else-branch (s_heap.assign(m, 0)) could allocate gigabytes of memory without MemTracker awareness.
Suggested fix — add a guard after computing m and n, consistent with the other functions:
constexprsize_tMAX_INPUT_LEN = 65535;
if (m > MAX_INPUT_LEN || n > MAX_INPUT_LEN) {
throwdoris::Exception(ErrorCode::INVALID_ARGUMENT,
"Input string too long for jaro_winkler, max {} bytes",
MAX_INPUT_LEN);
}| res = 0.0; | ||
| return; | ||
| } | ||
| const size_t m = s.size(), n = t.size(); |
There was a problem hiding this comment.
[MEDIUM] Missing input length guard.JaccardSimilarityOp has no input length validation. While its core algorithm is O(m log m + n log n) (less critical than Jaro-Winkler's quadratic), the heap path at line 539 (std::vector<uint16_t>(m - 1)) allocates 2 * (m-1) bytes untracked by MemTracker. For a STRING-type input near the ~2GB limit, this could attempt a ~4GB heap allocation.
For consistency with the other functions and to prevent OOM, consider adding a length guard:
constexprsize_tMAX_INPUT_LEN = 65535;
if (m > MAX_INPUT_LEN || n > MAX_INPUT_LEN) {
throwdoris::Exception(ErrorCode::INVALID_ARGUMENT,
"Input string too long for jaccard_similarity, max {} bytes",
MAX_INPUT_LEN);
}doris-robot
commented
Mar 3, 2026
TPC-H: Total hot run time: 28831 ms |
doris-robot
commented
Mar 3, 2026
TPC-DS: Total hot run time: 183808 ms |
hello-stephen
commented
Mar 3, 2026
FE Regression Coverage ReportIncrement line coverage |
doris-robot
commented
Mar 3, 2026
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Mar 3, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
…ions Add five new built-in scalar functions for calculating string similarity/distance: - levenshtein(str1, str2) -> INT - damerau_levenshtein(str1, str2) -> INT - jaro_winkler(str1, str2) -> DOUBLE Jaro-Winkler similarity in [0.0, 1.0] - cosine_similarity(str1, str2) -> DOUBLE Cosine similarity based on byte-frequency vectors (256-dim, stack-allocated). - jaccard_similarity(str1, str2) -> DOUBLE All functions are implemented in C++ with corresponding FE scalar function classes and visitor methods.
54ddedd to
a584a8eComparepuranjay2597
commented
Mar 4, 2026
Addressed the AI code review comments: Added input length guards to jaro_winkler and jaccard_similarity Could you please run run buildall again? |
| static constexpr auto name = "levenshtein"; | ||
| }; | ||
| struct LevenshteinOp { |
There was a problem hiding this comment.
It seems like you are processing strings byte by byte, but I think we need to be compatible with UTF-8 here, just like Hive does：
select levenshtein('你好', '世界')
+------+
| _c0 |
+------+
| 2 |
+------+
Check if it is all ASCII, if not, then proceed with the UTF-8 path, you can refer to FunctionLeft
ditto in another three functions
| static constexpr auto name = "jaro_winkler"; | ||
| }; | ||
| struct JaroWinklerOp { |
There was a problem hiding this comment.
Add comments to clearly explain what each section is calculating.
| ++k; | ||
| } | ||
| } | ||
| const double jaro = (matches / static_cast<double>(m) + matches / static_cast<double>(n) + |
There was a problem hiding this comment.
implement a new function JARO, move the above implementation into JARO, and call it directly here
| MAX_INPUT_LEN); | ||
| } | ||
| auto collect = [](const std::string_view& str, uint16_t* out) -> size_t { |
| const size_t tc = collect(t, t_bg.data()); | ||
| merge_count(s_bg.data(), sc, t_bg.data(), tc, intersect, union_sz); | ||
| } | ||
| res = (union_sz > 0) ? static_cast<double>(intersect) / union_sz : 1.0; |
There was a problem hiding this comment.
| res = (union_sz > 0) ? static_cast<double>(intersect) / union_sz : 1.0; | |
| res = static_cast<double>(intersect) / union_sz; |
already check s.size() < 2 || t.size() < 2 above
linrrzqqq
commented
Mar 7, 2026
impl fold constant in |
There was a problem hiding this comment.
more test, include column args, column and constant combination, and utf8 ......
| for (size_t i = 0; i + 1 < str.size(); ++i) | ||
| out[cnt++] = static_cast<uint16_t>((static_cast<uint16_t>((uint8_t)str[i]) << 8) | | ||
| (uint8_t)str[i + 1]); | ||
| std::sort(out, out + cnt); |
There was a problem hiding this comment.
try to use bitset to collect and do intersects/union
zclllyybb
commented
Apr 20, 2026
continue by #60412 |
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
puranjay2597
commented
Sep 2, 2026
GitHub does not allow reopening this PR because the branch was force-pushed after it was closed. I have addressed the review feedback (UTF-8 support, reduced duplication between jaro/jaro_winkler, reworked jaccard_similarity to match ClickHouse's stringJaccardIndex semantics, FE constant folding, expanded tests) and dropped levenshtein/damerau_levenshtein since they since landed separately (#60412, #65278). Continuing as #67436. |
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors).
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors).
What problem does this PR solve?
Adds 3 built-in scalar functions for fuzzy string matching and similarity scoring, useful for record deduplication, search ranking, and data quality workflows:
jaro(str1, str2)[0.0, 1.0]jaro_winkler(str1, str2)[0.0, 1.0](boosts strings sharing a common prefix)jaccard_similarity(str1, str2)[0.0, 1.0]over the sets of distinct characters of the two stringsAll functions accept
VARCHAR/STRINGinputs, propagate NULL, and support constant folding.Rebased and reworked since this PR was last open
This PR originally also proposed
levenshteinanddamerau_levenshtein. Both now already exist on master under different names (levenshteinvia #60412,damerau_levenshtein_distancevia #65278), so they've been dropped from this PR to avoid duplicating functionality — only the 3 functions above remain.The remaining functions have been reworked from the original submission to address review feedback:
jaro_winkler('你好世界', '你好世间')now compares by character. Implemented with an ASCII fast path plus a UTF-8-aware path, following the exact pattern established bylevenshtein/damerau_levenshtein_distance(VStringFunctions::get_utf8_char_offsets/utf8_char_equal).jarofunction containing the core Jaro algorithm;jaro_winklernow calls it directly instead of duplicating the matching/transposition logic.jaccard_similarityis now a character-set Jaccard index —|A ∩ B| / |A ∪ B|over the sets of distinct bytes (ASCII) or Unicode characters (UTF-8) — matching ClickHouse'sstringJaccardIndex(FunctionsStringDistance.cpp) instead of an unexplained bigram scheme. Uses astd::bitset<256>for the ASCII path per the reviewer's suggestion, and a hash set of UTF-8 characters otherwise.jaro/jaro_winkler/jaccard_similarityfold-constant implementations inStringArithmetic.java, matching the convention used bylevenshtein/damerau_levenshtein_distance/hamming_distance.jaccard_similarityagainst unboundedSTRINGinputs.function_string_test.cpp) and regression tests now cover column-vs-column, column-vs-constant (both directions), nullable columns, UTF-8, and over-length-input error cases, in addition to the constant-only cases.Release note
Add 3 built-in string similarity functions:
jaro,jaro_winkler,jaccard_similarity.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Release note
Add 3 built-in string similarity functions:
jaro,jaro_winkler,jaccard_similarity.Code Review