Uh oh!
There was an error while loading. Please reload this page.
[Feature](function) Support function DAMERAU_LEVENSHTEIN_DISTANCE - #65278
Merged
zclllyybb merged 3 commits intoJul 9, 2026
Merged
Conversation
linrrzqqq
requested review from
924060929, englefly, morrySnow and starocean999
as code ownersJuly 6, 2026 13:28
hello-stephen
commented
Jul 6, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
linrrzqqqforce-pushed
the
support-damerau-levenshtein-distance
branch
from
July 6, 2026 13:31
738ef39 to
e875c26Comparelinrrzqqq
commented
Jul 6, 2026
CollaboratorAuthor
/review |
Contributor
There was a problem hiding this comment.
Automated review completed for PR #65278.
I found no blocking correctness issues and no inline comments to submit.
Critical checkpoint conclusions:
- Goal and tests: The PR adds
damerau_levenshtein_distancepluslevenshtein_distance/edit_distancealiases. FE binding, FE constant folding, BE execution, and regression coverage are present for literal, vector/vector, vector/scalar, NULL, UTF-8, transposition, and oversized matrix cases. - Scope: The function changes are focused on the string-distance implementation, FE scalar registration, executable folding, benchmark coverage, and one regression suite.
- Concurrency, lifecycle, persistence, and configuration: Not applicable; this is a deterministic scalar function addition with no new shared state, config, persistence, RPC, or lifecycle path.
- FE/BE compatibility: FE and BE expose matching names and return
INT; aliases map consistently, and the FE/BE Damerau implementations use the same full recurrence and 16M-cell guard. - Parallel paths: Runtime BE execution and FE constant folding were checked. The scalar follows the existing
Levenshteinnullable/signature pattern and does not require optimizer, join, aggregate, or property-derivation changes. - Test result review: The added
.outrows match an independent full Damerau-Levenshtein calculation for the covered cases. Patch whitespace/reverse-apply hygiene passed. - Validation limits: I did not run FE/BE builds or regression tests because this checkout lacks
thirdparty/installed/bin/protoc.
User focus: No additional user-provided review focus was supplied.
Subagent conclusions:
optimizer-rewrite: no optimizer/rewrite candidate findings; convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.tests-session-config: no tests/session/config candidate findings; convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.- No candidates were accepted, dismissed as duplicates, or submitted as inline comments.
linrrzqqq
commented
Jul 7, 2026
CollaboratorAuthor
run buildall |
hello-stephen
commented
Jul 7, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Jul 7, 2026
Contributor
TPC-H: Total hot run time: 29778 ms |
hello-stephen
commented
Jul 7, 2026
Contributor
TPC-DS: Total hot run time: 179471 ms |
hello-stephen
commented
Jul 7, 2026
Contributor
ClickBench: Total hot run time: 24.97 s |
hello-stephen
commented
Jul 7, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jul 7, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jul 7, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
zclllyybb
previously approved these changes
Jul 8, 2026
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
linrrzqqq
commented
Jul 8, 2026
CollaboratorAuthor
run buildall |
linrrzqqq
commented
Jul 8, 2026
CollaboratorAuthor
/review |
Contributor
There was a problem hiding this comment.
Static review completed for PR 65278.
I reviewed the full changed-file set for the new damerau_levenshtein_distance function, the levenshtein_distance / edit_distance aliases, FE/Nereids registration and constant folding, BE vectorized execution, benchmark wiring, and regression expected output. I did not find a substantiated issue that needs an inline review comment.
Critical checkpoint conclusions:
- Goal/test coverage: the PR adds the FE and BE function surfaces plus regression coverage for constants, aliases, nulls, ASCII/UTF-8 values, vector-vector, vector-scalar, scalar-vector, and the matrix-size error path.
- Scope/focus: the implementation is focused on the string-distance function family and reuses the existing
levenshteinregistration and execution patterns. - Concurrency/lifecycle/config/persistence: no new concurrency, lifecycle, dynamic config, persistence, storage format, or transaction path was introduced.
- Compatibility/parallel paths: FE and BE register the new function consistently; the new aliases are registered in both FE and BE. The Nereids scalar class, visitor, and executable folding path are wired.
- Function correctness: representative expected results were cross-checked against full Damerau-Levenshtein semantics, including transposition cases and UTF-8 cases.
- Performance/resource behavior: the BE and FE paths both enforce the 16,777,216-cell matrix guard for Damerau-Levenshtein. No additional actionable performance finding was identified.
- Tests/style: added outputs match the added queries in the reviewed diff, and added patch lines had no trailing whitespace.
Validation limits: this runner is a shallow checkout without the base commit locally, and thirdparty/installed / thirdparty/installed/bin/protoc are absent, so I used GitHub PR metadata/patches as the authoritative diff and did not run FE/BE builds or regression tests.
Subagent conclusions:
optimizer-rewritereviewed optimizer/rewrite, scalar registration, visitor integration, FE executable folding, translator path, alias behavior, and FE/BE semantic parity. It reportedNO_NEW_VALUABLE_FINDINGS.tests-session-configreviewed regression tests, expected outputs, session/config propagation, compatibility, CI/style basics, and build/test prerequisites. It reportedNO_NEW_VALUABLE_FINDINGS.- Final convergence round 1 used the current shared ledger and empty proposed inline comment set; both live subagents replied
NO_NEW_VALUABLE_FINDINGS.
User focus: no additional user-provided focus points were present.
hello-stephen
commented
Jul 8, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
linrrzqqq
commented
Jul 8, 2026
CollaboratorAuthor
run buildall |
hello-stephen
commented
Jul 8, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Jul 8, 2026
Contributor
TPC-H: Total hot run time: 29616 ms |
linrrzqqq
commented
Jul 8, 2026
CollaboratorAuthor
/review |
hello-stephen
commented
Jul 8, 2026
Contributor
TPC-DS: Total hot run time: 181061 ms |
hello-stephen
commented
Jul 8, 2026
Contributor
ClickBench: Total hot run time: 24.97 s |
Contributor
There was a problem hiding this comment.
Automated review completed for PR 65278.
I did not find a new, distinct correctness issue that should be submitted as an inline review comment. The two ordering concerns in BuiltinScalarFunctions.java are already covered by existing inline threads:
DamerauLevenshteinDistance.classordering beforeDatabase.classLevenshtein.classordering afterLength.class
Critical checkpoint conclusions:
- Goal and coverage: the PR adds
damerau_levenshtein_distance, Levenshtein aliases, FE/Nereids registration, BE execution, benchmark coverage, and regression coverage. Tests cover constant folding, nullable inputs, table/vector paths, UTF-8 transpositions, aliases, and the large matrix error path. - Scope: the implementation is focused on string distance functions and their registration/tests.
- FE/BE consistency: BE registration, FE
BuiltinScalarFunctions, Nereids scalar class, visitor, and executable constant-folding path all use matching names and compatible integer return semantics. - Parallel paths: scalar/scalar constant folding and BE vector/vector, vector/scalar, and scalar/vector paths were checked. The new
Statuserrors from the BE matrix-size guard are propagated byFunctionBinaryToType. - Concurrency and lifecycle: no new runtime shared mutable state, locks, async paths, persistence, edit logs, storage formats, or config items are introduced.
- Memory/performance: the Damerau implementation has an explicit 16M-cell matrix guard, uses allocator-aware containers for the new BE-owned buffers, and avoids the slower UTF-8 direct-comparison inner loop.
- Observability: no additional logs or metrics appear necessary for this scalar function.
- Tests: regression cases and expected output are present, but I did not run them locally because this runner is missing
.worktree_initialized,thirdparty/installed, andthirdparty/installed/bin/protoc.
User focus: no additional user-provided review focus was present.
Subagent conclusions:
optimizer-rewrite: no candidates appended; final convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.tests-session-config: no candidates appended; final convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.- Main duplicate suppression: MAIN-DUP-001 and MAIN-DUP-002 were not resubmitted because they duplicate existing inline review threads.
- Main dismissed checks: MAIN-DISMISS-001 and MAIN-DISMISS-002 were dismissed with code evidence for FE/BE matrix-limit and UTF-8 folding/runtime consistency.
hello-stephen
commented
Jul 8, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jul 8, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
1 similar comment
hello-stephen
commented
Jul 8, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
924060929
approved these changes
Jul 9, 2026
Contributor
PR approved by at least one committer and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
9 tasks
puranjay2597 pushed a commit
to puranjay2597/doris
that referenced
this pull request
Sep 2, 2026
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
5 tasks
puranjay2597 pushed a commit
to puranjay2597/doris
that referenced
this pull request
Sep 2, 2026
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
puranjay2597 pushed a commit
to puranjay2597/doris
that referenced
this pull request
Sep 2, 2026
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors).
puranjay2597 pushed a commit
to puranjay2597/doris
that referenced
this pull request
Sep 2, 2026
…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors).
puranjay2597 pushed a commit
to puranjay2597/doris
that referenced
this pull request
Sep 3, 2026
…rd with BE Reviewed the merged levenshtein/hamming_distance (apache#60412) and damerau_levenshtein_distance (apache#65278, apache#66236) PRs for consistency with this one. Found and fixed a real FE/BE divergence in the process: - The FE constant-fold length guard checked Java code point count, while the BE guard checks UTF-8 byte length. For multi-byte input the two disagree (e.g. ~30000 3-byte characters is under the FE's 65535 code-point cap but over BE's 65535-byte cap), so a literal expression could fold successfully on FE while the same value would be rejected by BE if read from a column. FE now measures UTF-8 bytes too, matching function_string_similarity.cpp exactly (mirrors the fix pattern in apache#64881, "Align constant folding with BE results"). - Added regression coverage for astral-plane (surrogate-pair/4-byte UTF-8) characters on all three functions, matching the precedent set for RIGHT/INSTR in apache#64881.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release note
Support function
DAMERAU_LEVENSHTEIN_DISTANCEdoc: apache/doris-website#3971