Add levenshtein and hamming_distance functions by KazeBox33 · Pull Request #60412 · apache/doris

KazeBox33 · 2026-02-01T16:11:29Z

What problem does this PR solve?

Related Issue: #48203
Related PR: #57144 (reference)
Problem Summary: support levenshtein (Hive) and hamming_distance (Trino/Presto).

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
  - ./run-regression-test.sh --run -d regression-test/suites/query_p0/sql_functions/string_functions -s string_functions_all
  - ./run-regression-test.sh --run -d regression-test/suites/nereids_p0/sql_functions/string_functions -s test_string_function
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes. Add document PR link here: [doc] add docs for levenshtein and hamming_distance doris-website#3324

Thearas · 2026-02-01T16:11:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR adds support for two string distance functions: levenshtein (Hive compatibility) and hamming_distance (Trino/Presto compatibility). These functions compute edit distance and character-difference distance between strings respectively, with proper UTF-8 character handling.

Changes:

Implemented levenshtein function that calculates the minimum number of single-character edits needed to transform one string into another
Implemented hamming_distance function that counts character differences between equal-length strings
Added comprehensive test coverage for both functions including edge cases, NULL handling, and UTF-8 character support

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test_string_all.groovy	Added regression tests for levenshtein and hamming_distance functions with various input scenarios
test_string_function.groovy	Added nereids query tests covering edge cases and error conditions for both functions
test_string_all.out	Expected output for levenshtein and hamming_distance test cases
test_string_function.out	Expected output for nereids function tests
ScalarFunctionVisitor.java	Added visitor methods for HammingDistance and Levenshtein function nodes
Levenshtein.java	Implemented Levenshtein scalar function class with signature definitions
HammingDistance.java	Implemented HammingDistance scalar function class with signature definitions
StringArithmetic.java	Added executable implementations for levenshtein and hamming_distance with dynamic programming algorithm
BuiltinScalarFunctions.java	Registered new functions in the builtin scalar functions registry
simple_function_factory.h	Added function registration declarations for backend implementations
function_string.cpp	Minor whitespace cleanup
function_levenshtein.cpp	Backend C++ implementation of levenshtein with UTF-8 support
function_hamming_distance.cpp	Backend C++ implementation of hamming_distance with UTF-8 support and length validation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zclllyybb · 2026-02-02T12:01:52Z

and remember to format your code, add more cases which query from table

zclllyybb · 2026-03-14T15:28:46Z

run buildall

zclllyybb · 2026-03-14T15:28:49Z

/review

github-actions

Code Review Summary for PR #60412: Add levenshtein and hamming_distance functions

Overall this is a well-structured PR that adds two useful string distance functions. The FE-BE type consistency is correct, the constant folding implementations match the BE implementations, and the test coverage is reasonable. Below are the issues found, organized by the critical checkpoint conclusions.

Critical Checkpoint Conclusions

1. Goal and correctness: The PR adds levenshtein (returns Int32) and hamming_distance (returns Int64) string distance functions with UTF-8 support. The core algorithms are correct. FE constant folding, FE scalar function classes, BE implementations, and registration are all present.

2. Modification scope: The change is focused - 13 files, all related to the two new functions. Minor cosmetic changes in function_string.cpp (blank line changes) are unnecessary but harmless.

3. Concurrency: No concurrency concerns - these are pure stateless scalar functions.

4. Lifecycle management: No special lifecycle concerns.

5. Configuration items: None added. N/A.

6. Incompatible changes: None. New functions only.

7. Parallel code paths:hamming_distance uses a custom FunctionBinaryStringToTypeWithNull wrapper class instead of the standard FunctionBinaryToType used by levenshtein. This is necessary because hamming_distance can return Status::InvalidArgument for unequal-length strings, and the custom class provides proper row-level null handling with explicit nullable return type control. This is an acceptable design choice.

8. Special conditional checks: The string_ref_at helper in both files contains defensive if checks that violate AGENTS.md coding standards. See inline comments.

9. Test coverage: Good regression test coverage for both functions with ASCII, UTF-8, NULL, empty string, and table-based tests. Error case for hamming_distance is tested. However, there is an orphaned .out entry. See inline comments.

10. Observability: N/A for pure scalar functions.

11. Transaction/persistence: N/A.

12. FE-BE variable passing: FE-BE types are consistent (levenshtein: Int32/Int32, hamming_distance: Int64/Int64). PropagateNullable in FE aligns with BE null handling for both functions.

13. Performance: The levenshtein function has O(m*n) time complexity with no input size limit, which is a potential DoS concern for very large strings. See inline comments.

14. Other issues: Alias levenshtein_distance (common Hive name) is not registered - this may be intentional but worth considering for compatibility.

doris-robot · 2026-03-14T16:06:15Z

TPC-H: Total hot run time: 26861 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68d4ca6033fa03d84cbcccd04c127de1e003f48d, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17670	4442	4323	4323
q2	q3	10637	822	524	524
q4	4681	369	251	251
q5	7542	1202	1017	1017
q6	174	172	148	148
q7	797	859	671	671
q8	9301	1476	1343	1343
q9	4919	4808	4726	4726
q10	6232	1904	1664	1664
q11	471	261	261	261
q12	684	579	470	470
q13	18056	2918	2179	2179
q14	233	237	221	221
q15	q16	729	724	672	672
q17	713	842	430	430
q18	6011	5356	5255	5255
q19	1141	1003	609	609
q20	541	493	383	383
q21	4513	1817	1388	1388
q22	348	490	326	326
Total cold run time: 95393 ms
Total hot run time: 26861 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4792	4641	4583	4583
q2	q3	3871	4331	3812	3812
q4	863	1186	777	777
q5	4115	4369	4312	4312
q6	185	176	145	145
q7	1738	1632	1495	1495
q8	2509	2729	2519	2519
q9	7675	7414	7417	7414
q10	3772	3972	3581	3581
q11	506	431	428	428
q12	494	597	436	436
q13	2634	3139	2251	2251
q14	284	286	272	272
q15	q16	739	773	700	700
q17	1158	1407	1400	1400
q18	7210	6738	6576	6576
q19	872	863	921	863
q20	2016	2118	2050	2050
q21	3971	3477	3859	3477
q22	511	412	386	386
Total cold run time: 49915 ms
Total hot run time: 47477 ms

doris-robot · 2026-03-14T16:16:59Z

TPC-DS: Total hot run time: 167910 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68d4ca6033fa03d84cbcccd04c127de1e003f48d, data reload: false
query5	4363	650	523	523
query6	329	221	210	210
query7	4235	497	267	267
query8	344	257	239	239
query9	8703	2709	2731	2709
query10	504	387	333	333
query11	7000	5109	4885	4885
query12	176	126	124	124
query13	1248	457	337	337
query14	5776	3735	3439	3439
query14_1	2787	2795	2816	2795
query15	203	196	175	175
query16	1000	486	458	458
query17	1120	717	615	615
query18	2445	463	350	350
query19	222	211	186	186
query20	131	127	127	127
query21	214	132	110	110
query22	13223	14056	14624	14056
query23	16423	15859	15720	15720
query23_1	15647	15541	15331	15331
query24	7386	1636	1236	1236
query24_1	1239	1239	1239	1239
query25	559	483	443	443
query26	1231	280	150	150
query27	2754	495	298	298
query28	4475	1854	1837	1837
query29	886	590	502	502
query30	305	232	199	199
query31	1027	936	874	874
query32	88	72	74	72
query33	523	343	300	300
query34	899	926	521	521
query35	654	675	600	600
query36	1079	1099	964	964
query37	142	98	89	89
query38	2934	2943	2863	2863
query39	852	848	848	848
query39_1	811	804	793	793
query40	230	150	136	136
query41	62	57	58	57
query42	271	264	257	257
query43	252	245	215	215
query44 query45	200	184	178	178
query46	916	1008	611	611
query47	2094	2139	2048	2048
query48	318	321	229	229
query49	623	460	385	385
query50	705	281	211	211
query51	4091	4089	4028	4028
query52	253	271	252	252
query53	302	334	281	281
query54	285	273	252	252
query55	90	85	78	78
query56	322	317	306	306
query57	1728	1787	1813	1787
query58	279	275	267	267
query59	2819	2962	2744	2744
query60	339	333	323	323
query61	154	143	150	143
query62	620	581	542	542
query63	320	280	273	273
query64	5088	1266	975	975
query65 query66	1454	452	354	354
query67	24354	24429	24204	24204
query68 query69	425	317	289	289
query70	936	890	955	890
query71	343	303	297	297
query72	2779	2679	2158	2158
query73	551	564	323	323
query74	9633	9585	9402	9402
query75	2912	2752	2468	2468
query76	2273	1083	702	702
query77	376	399	311	311
query78	10896	11058	10435	10435
query79	1152	806	557	557
query80	903	636	542	542
query81	520	255	227	227
query82	1357	146	120	120
query83	344	276	243	243
query84	251	119	99	99
query85	900	494	431	431
query86	417	340	289	289
query87	3132	3144	3027	3027
query88	3589	2675	2666	2666
query89	444	377	343	343
query90	1840	177	175	175
query91	168	169	133	133
query92	75	73	74	73
query93	939	882	503	503
query94	513	321	305	305
query95	579	410	315	315
query96	641	541	235	235
query97	2476	2480	2407	2407
query98	234	221	232	221
query99	1025	984	936	936
Total cold run time: 249265 ms
Total hot run time: 167910 ms

hello-stephen · 2026-03-14T16:41:52Z

FE UT Coverage Report

Increment line coverage 3.03% (2/66) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-03-14T17:14:46Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.65% (19733/37482)
Line Coverage	36.25% (184414/508725)
Region Coverage	32.39% (142433/439808)
Branch Coverage	33.55% (62204/185420)

hello-stephen · 2026-03-14T18:34:20Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.64% (26281/36687)
Line Coverage	54.41% (275861/507003)
Region Coverage	51.53% (228722/443856)
Branch Coverage	53.06% (98633/185902)

KazeBox33 · 2026-03-15T11:16:48Z

run buildall

doris-robot · 2026-03-15T11:54:18Z

TPC-H: Total hot run time: 26915 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f5e716f67e18a0cec437e795b6bfcdfc7bdb6e0, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17660	4447	4272	4272
q2	q3	10642	798	517	517
q4	4711	363	255	255
q5	7826	1203	1034	1034
q6	198	180	148	148
q7	812	836	668	668
q8	10402	1481	1365	1365
q9	5566	4773	4792	4773
q10	6354	1940	1632	1632
q11	492	266	232	232
q12	738	570	468	468
q13	18036	2925	2171	2171
q14	230	227	226	226
q15	q16	736	723	663	663
q17	733	846	447	447
q18	5872	5351	5240	5240
q19	1336	983	600	600
q20	541	486	381	381
q21	4575	2019	1514	1514
q22	367	347	309	309
Total cold run time: 97827 ms
Total hot run time: 26915 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4754	4648	4531	4531
q2	q3	3894	4342	3803	3803
q4	875	1206	790	790
q5	4077	4357	4396	4357
q6	192	179	149	149
q7	1797	1630	1517	1517
q8	2502	2752	2563	2563
q9	7501	7499	7425	7425
q10	3700	3912	3672	3672
q11	514	486	406	406
q12	474	615	443	443
q13	2842	3453	2335	2335
q14	283	305	285	285
q15	q16	743	768	755	755
q17	1219	1387	1426	1387
q18	7274	6761	6725	6725
q19	952	878	852	852
q20	2112	2175	2035	2035
q21	4189	3490	3364	3364
q22	487	416	388	388
Total cold run time: 50381 ms
Total hot run time: 47782 ms

doris-robot · 2026-03-15T12:05:05Z

TPC-DS: Total hot run time: 168305 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f5e716f67e18a0cec437e795b6bfcdfc7bdb6e0, data reload: false
query5	4337	629	519	519
query6	327	225	219	219
query7	4207	473	273	273
query8	347	240	226	226
query9	8747	2729	2727	2727
query10	504	407	356	356
query11	6988	5045	4857	4857
query12	187	128	124	124
query13	1261	443	355	355
query14	5700	3705	3443	3443
query14_1	2838	2815	2799	2799
query15	210	198	179	179
query16	978	455	444	444
query17	894	717	631	631
query18	2448	446	350	350
query19	221	217	182	182
query20	134	132	129	129
query21	212	135	112	112
query22	13242	14310	14608	14310
query23	16250	15646	15589	15589
query23_1	15651	15784	15509	15509
query24	7133	1611	1194	1194
query24_1	1214	1218	1234	1218
query25	537	478	418	418
query26	1271	265	153	153
query27	2784	476	287	287
query28	4526	1854	1842	1842
query29	822	557	475	475
query30	294	222	186	186
query31	1012	964	882	882
query32	83	72	70	70
query33	507	338	274	274
query34	951	881	529	529
query35	637	667	596	596
query36	1112	1147	1018	1018
query37	130	91	82	82
query38	2964	2885	2845	2845
query39	861	823	802	802
query39_1	801	797	789	789
query40	229	153	137	137
query41	62	62	60	60
query42	251	250	257	250
query43	245	248	228	228
query44 query45	194	241	191	191
query46	886	998	614	614
query47	2126	2156	2070	2070
query48	322	311	223	223
query49	623	481	397	397
query50	671	280	209	209
query51	4069	4061	3998	3998
query52	261	264	256	256
query53	296	338	280	280
query54	298	276	257	257
query55	99	89	79	79
query56	317	330	309	309
query57	1940	1776	1619	1619
query58	285	271	263	263
query59	2767	2926	2758	2758
query60	337	336	320	320
query61	154	156	155	155
query62	628	578	532	532
query63	307	277	273	273
query64	5047	1275	984	984
query65 query66	1472	448	365	365
query67	24133	24151	24116	24116
query68 query69	396	312	278	278
query70	974	971	951	951
query71	337	307	296	296
query72	2912	2875	2595	2595
query73	543	550	323	323
query74	9594	9556	9385	9385
query75	2862	2739	2522	2522
query76	2296	1038	664	664
query77	371	432	341	341
query78	10907	11000	10429	10429
query79	2931	797	580	580
query80	1751	628	538	538
query81	576	259	226	226
query82	993	148	118	118
query83	337	265	244	244
query84	301	121	98	98
query85	892	470	436	436
query86	501	291	292	291
query87	3131	3063	3061	3061
query88	3534	2623	2644	2623
query89	425	366	342	342
query90	2237	173	166	166
query91	165	160	143	143
query92	88	76	69	69
query93	2433	838	497	497
query94	650	313	300	300
query95	585	337	315	315
query96	642	513	238	238
query97	2487	2518	2412	2412
query98	239	222	219	219
query99	1032	995	905	905
Total cold run time: 252881 ms
Total hot run time: 168305 ms

hello-stephen · 2026-03-15T12:23:34Z

FE UT Coverage Report

Increment line coverage 3.03% (2/66) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-03-15T12:54:32Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.63% (19726/37479)
Line Coverage	36.21% (184217/508686)
Region Coverage	32.35% (142166/439444)
Branch Coverage	33.55% (62171/185335)

### What problem does this PR solve? Issue Number: None Related PR: apache#60412 Problem Summary: Trim the trailing storage zero from const scalar string arguments in the BE levenshtein const-scalar paths so they match vector-vector execution and FE constant folding semantics. ### Release note None ### Check List (For Author) - Test: No need to test (local BE build environment is unavailable in this workspace; existing regression cases already cover the const-scalar paths adjusted here) - Behavior changed: Yes (BE const-scalar levenshtein no longer counts the trailing storage zero as a real character) - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#60412 Problem Summary: Remove whole-file compile_check_avoid wrappers from the new hamming_distance and levenshtein BE implementations so normal conversion warnings remain enabled for new code. ### Release note None ### Check List (For Author) - Test: No need to test (local thirdparty headers required for BE syntax build are unavailable in this workspace; formatting and diff checks were run) - Behavior changed: No - Does this need documentation: No

linrrzqqq · 2026-05-27T02:30:40Z

run buildall

linrrzqqq · 2026-05-27T02:30:46Z

/review

github-actions

Automated review summary for PR 60412 at 60539aa.

I did not add new inline comments because the remaining concerns I found are already covered by existing review threads and should not be duplicated. In particular, please resolve the already-open discussions around vstring_function.h header self-containment, levenshtein FE/BE constant-folding cost guards, debug_skip_fold_constant test session-state restoration, and string trailing-zero handling/access consistency.

Critical checkpoint conclusions:

Goal/test proof: the PR adds levenshtein and hamming_distance in FE/BE with regression coverage, including Nereids and negative hamming-distance cases. Existing threads still cover gaps/risks in constant-fold cost and test session state.
Scope: the implementation is focused on the two new string functions and their registrations/tests.
Concurrency/lifecycle: no new shared mutable state, threads, locks, globals with non-trivial cross-TU initialization, or lifecycle-sensitive ownership were introduced.
Configuration/compatibility: no new config items, storage formats, persistence, transaction paths, or FE-BE protocol fields were added.
Parallel paths: both BE execution and FE/Nereids binding/folding paths were added; existing review context already calls out remaining coverage/behavior concerns.
Error handling: hamming_distance unequal-length errors are propagated through Status in BE and AnalysisException in FE; no ignored Status found in the reviewed diff.
Memory/performance: the Levenshtein DP remains O(m*n) and existing review threads already flag lack of guard for large constant/runtime inputs.
Observability: no new distributed or background operational path requiring new metrics/logging.
User focus: no additional user-provided review focus was present.

No additional distinct issues found beyond the known review context.

hello-stephen · 2026-05-27T04:18:14Z

TPC-H: Total hot run time: 32172 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60539aa845bcce4d1ad9d9299c2f3b8a8828e7f8, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17601	4032	4011	4011
q2	q3	10790	1453	837	837
q4	4700	485	357	357
q5	7628	2310	2203	2203
q6	250	183	141	141
q7	999	831	650	650
q8	9401	1810	1758	1758
q9	5251	5049	4988	4988
q10	6417	2191	1892	1892
q11	439	279	261	261
q12	640	440	302	302
q13	18186	3353	2799	2799
q14	272	262	249	249
q15	q16	832	782	709	709
q17	1001	991	955	955
q18	6811	5605	5619	5605
q19	1304	1417	1222	1222
q20	573	467	293	293
q21	6363	2884	2629	2629
q22	461	368	311	311
Total cold run time: 99919 ms
Total hot run time: 32172 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4768	4701	4754	4701
q2	q3	4971	5315	4689	4689
q4	2145	2220	1436	1436
q5	5082	4705	4632	4632
q6	247	191	142	142
q7	1879	1765	1568	1568
q8	2424	2194	2093	2093
q9	7873	7408	7446	7408
q10	4755	4691	4222	4222
q11	534	382	352	352
q12	735	749	524	524
q13	3024	3430	2807	2807
q14	273	279	263	263
q15	q16	677	699	619	619
q17	1286	1269	1251	1251
q18	7100	6988	6798	6798
q19	1143	1115	1136	1115
q20	2222	2245	1939	1939
q21	5310	4639	4479	4479
q22	531	465	424	424
Total cold run time: 56979 ms
Total hot run time: 51462 ms

hello-stephen · 2026-05-27T04:29:25Z

TPC-DS: Total hot run time: 172191 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60539aa845bcce4d1ad9d9299c2f3b8a8828e7f8, data reload: false
query5	4325	654	523	523
query6	360	225	210	210
query7	4244	564	316	316
query8	351	239	231	231
query9	8809	4024	4010	4010
query10	471	357	315	315
query11	5784	2598	2285	2285
query12	199	134	127	127
query13	1341	634	444	444
query14	6055	5458	5168	5168
query14_1	4472	4494	4428	4428
query15	218	211	197	197
query16	1008	458	448	448
query17	1160	750	628	628
query18	2599	507	370	370
query19	220	214	172	172
query20	147	138	137	137
query21	225	143	118	118
query22	13605	13660	13409	13409
query23	17426	16613	16152	16152
query23_1	16398	16370	16296	16296
query24	7603	1796	1362	1362
query24_1	1412	1327	1358	1327
query25	598	533	447	447
query26	1313	319	182	182
query27	2674	521	341	341
query28	4413	2018	2011	2011
query29	1024	644	525	525
query30	322	231	199	199
query31	1125	1089	951	951
query32	99	79	74	74
query33	566	383	295	295
query34	1234	1135	653	653
query35	782	811	689	689
query36	1427	1426	1296	1296
query37	154	103	92	92
query38	3208	3166	3069	3069
query39	926	938	920	920
query39_1	866	883	882	882
query40	235	152	128	128
query41	67	61	62	61
query42	107	110	106	106
query43	339	344	293	293
query44 query45	220	206	206	206
query46	1144	1206	720	720
query47	2382	2431	2243	2243
query48	402	400	289	289
query49	636	488	400	400
query50	1016	346	251	251
query51	4306	4346	4305	4305
query52	103	106	95	95
query53	251	273	208	208
query54	308	267	251	251
query55	97	92	92	92
query56	315	299	339	299
query57	1443	1431	1353	1353
query58	307	268	267	267
query59	1586	1687	1446	1446
query60	325	325	315	315
query61	164	156	163	156
query62	749	648	582	582
query63	248	194	207	194
query64	2423	811	659	659
query65 query66	1695	495	358	358
query67	29754	29657	29601	29601
query68 query69	480	347	307	307
query70	1029	1056	966	966
query71	311	275	264	264
query72	3054	2750	2426	2426
query73	1000	739	443	443
query74	5105	4929	4776	4776
query75	2693	2600	2276	2276
query76	2289	1138	781	781
query77	401	397	321	321
query78	12658	12486	11857	11857
query79	1506	1098	742	742
query80	645	531	465	465
query81	451	279	253	253
query82	1380	160	123	123
query83	359	279	247	247
query84	273	148	113	113
query85	900	580	458	458
query86	409	323	324	323
query87	3443	3401	3268	3268
query88	3612	2735	2734	2734
query89	447	402	344	344
query90	1973	185	193	185
query91	181	165	144	144
query92	81	76	71	71
query93	1500	1506	853	853
query94	537	367	308	308
query95	691	485	348	348
query96	1117	762	353	353
query97	2754	2774	2620	2620
query98	245	237	233	233
query99	1174	1145	1031	1031
Total cold run time: 255258 ms
Total hot run time: 172191 ms

hello-stephen · 2026-05-27T05:43:24Z

BE Regression && UT Coverage Report

Increment line coverage 97.54% (397/407) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.85% (28106/38057)
Line Coverage	57.82% (305424/528260)
Region Coverage	55.09% (256180/465056)
Branch Coverage	56.56% (110591/195528)

hello-stephen · 2026-05-27T05:50:29Z

FE Regression Coverage Report

Increment line coverage 36.05% (62/172) 🎉
Increment coverage report
Complete coverage report

linrrzqqq

LGTM

1

github-actions · 2026-05-29T02:59:04Z

PR approved by at least one committer and no changes requested.

## Versions - [ ] dev - [x] 4.x - [ ] 3.x - [ ] 2.1 or older (not covered by version/language sync gate) ## Languages - [x] Chinese - [x] English - [ ] Japanese candidate translation needed ## Docs Checklist - [x] Checked by AI - [ ] Test Cases Built - [x] Updated required version and language counterparts, or explained why not - [x] If only one language changed, confirmed whether source/translation counterparts need sync Related PR: apache/doris#60412

Related Issue: apache#48203 Related PR: apache#57144 (reference) Problem Summary: support levenshtein (Hive) and hamming_distance (Trino/Presto).

…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ing functions Rebuilds the string-similarity functions proposed in apache#60799 on top of current master, addressing prior review feedback: - levenshtein and damerau_levenshtein are dropped: both now exist on master (apache#60412, apache#65278) under levenshtein/damerau_levenshtein_distance, so keeping ours would only collide. - All three functions get full UTF-8 support (ASCII fast path + character- aware path via VStringFunctions::get_utf8_char_offsets/utf8_char_equal), matching the pattern established by levenshtein/damerau_levenshtein_distance instead of operating on raw bytes. - jaro_winkler now shares its Jaro computation with the new jaro function instead of duplicating the matching/transposition logic. - jaccard_similarity is redefined as a character-set Jaccard index (bitset for the ASCII path, hash set of UTF-8 characters otherwise), matching ClickHouse's stringJaccardIndex semantics, rather than an unexplained byte-bigram scheme. - Added FE constant-folding (StringArithmetic.java) for all three functions, and BE unit tests plus expanded regression coverage (column/constant combinations, nullable columns, UTF-8, over-length-input errors).

…rd with BE Reviewed the merged levenshtein/hamming_distance (apache#60412) and damerau_levenshtein_distance (apache#65278, apache#66236) PRs for consistency with this one. Found and fixed a real FE/BE divergence in the process: - The FE constant-fold length guard checked Java code point count, while the BE guard checks UTF-8 byte length. For multi-byte input the two disagree (e.g. ~30000 3-byte characters is under the FE's 65535 code-point cap but over BE's 65535-byte cap), so a literal expression could fold successfully on FE while the same value would be rejected by BE if read from a column. FE now measures UTF-8 bytes too, matching function_string_similarity.cpp exactly (mirrors the fix pattern in apache#64881, "Align constant folding with BE results"). - Added regression coverage for astral-plane (surrogate-pair/4-byte UTF-8) characters on all three functions, matching the precedent set for RIGHT/INSTR in apache#64881.

KazeBox33 requested a review from zclllyybb as a code owner February 1, 2026 16:11

zclllyybb self-assigned this Feb 1, 2026

zclllyybb requested a review from Copilot February 2, 2026 11:44

Copilotstarted reviewing on behalf of zclllyybb February 2, 2026 11:45 View session

CopilotAI reviewed Feb 2, 2026
View reviewed changes

Comment threadregression-test/suites/query_p0/sql_functions/string_functions/test_string_all.groovy Outdated
Comment threadregression-test/suites/query_p0/sql_functions/string_functions/test_string_all.groovy Outdated

zclllyybb reviewed Feb 2, 2026
View reviewed changes

linrrzqqq reviewed Feb 3, 2026
View reviewed changes

Comment threadbe/src/vec/functions/function_levenshtein.cpp Outdated

KazeBox33 requested a review from zclllyybb March 2, 2026 16:25

KazeBox33 force-pushed the feature/levenshtein-hamming branch from fb0c27f to 5ccb369Compare March 14, 2026 12:31

github-actionsBot reviewed Mar 14, 2026
View reviewed changes

zclllyybb reviewed Mar 14, 2026
View reviewed changes

Comment threadbe/src/exprs/function/function_hamming_distance.cpp Outdated

linrrzqqq reviewed Mar 14, 2026
View reviewed changes

Comment threadbe/src/exprs/function/function_hamming_distance.cpp Outdated

linrrzqqq reviewed Mar 14, 2026
View reviewed changes

Comment threadbe/src/exprs/function/function_levenshtein.cpp Outdated

linrrzqqq reviewed Mar 14, 2026
View reviewed changes

Comment threadbe/src/exprs/function/function_levenshtein.cpp Outdated

KazeBox33 marked this pull request as draft March 15, 2026 09:07

KazeBox33 marked this pull request as ready for review March 15, 2026 11:16

KazeBox33 added 2 commits May 26, 2026 23:25

KazeBox33 force-pushed the feature/levenshtein-hamming branch from 936e5cc to 60539aaCompare May 26, 2026 15:35

github-actionsBot reviewed May 27, 2026
View reviewed changes

linrrzqqq approved these changes May 29, 2026
View reviewed changes

zclllyybb approved these changes May 29, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 29, 2026

zclllyybb merged commit 113fd2d into apache:masterMay 29, 2026
31 of 32 checks passed

zclllyybb added the dev/4.1.x label May 29, 2026

github-actionsBot added the dev/4.1.x-conflict label May 29, 2026

KazeBox33 mentioned this pull request May 29, 2026
Fix 4x hamming distance version note apache/doris-website#3844
Merged
11 tasks

KazeBox33 mentioned this pull request Jun 9, 2026
branch-4.1: Add levenshtein and hamming_distance functions #64316
Closed

yiguolei removed dev/4.1.x dev/4.1.x-conflict labels Jun 10, 2026

puranjay2597 mentioned this pull request Sep 2, 2026
[feature](function) Add jaro, jaro_winkler and jaccard_similarity string functions #67436
Open
5 tasks

Uh oh!

Conversation

KazeBox33 commented Feb 1, 2026

Uh oh!

Thearas commented Feb 1, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zclllyybb commented Feb 2, 2026

Uh oh!

Uh oh!

zclllyybb commented Mar 14, 2026

Uh oh!

zclllyybb commented Mar 14, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Code Review Summary for PR #60412: Add levenshtein and hamming_distance functions

Critical Checkpoint Conclusions

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Mar 14, 2026

Uh oh!

doris-robot commented Mar 14, 2026

Uh oh!

hello-stephen commented Mar 14, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Mar 14, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Mar 14, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KazeBox33 commented Mar 15, 2026

Uh oh!

doris-robot commented Mar 15, 2026

Uh oh!

doris-robot commented Mar 15, 2026

Uh oh!

hello-stephen commented Mar 15, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Mar 15, 2026

BE UT Coverage Report

Uh oh!

linrrzqqq commented May 27, 2026

Uh oh!

linrrzqqq commented May 27, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented May 27, 2026

Uh oh!

hello-stephen commented May 27, 2026

Uh oh!

hello-stephen commented May 27, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented May 27, 2026

FE Regression Coverage Report