[fix](function) Align constant folding with BE results by morrySnow · Pull Request #64881 · apache/doris

morrySnow · 2026-06-26T06:30:56Z

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: FE constant folding produced results different from BE execution for several string, URL, time, and floating-point expressions. This PR aligns the FE executable folding paths with BE behavior and adds testFoldConst coverage for the mismatched expressions.

Mismatched expressions

Expression	FE result before this PR	BE result	Root cause
`CONCAT_WS(',', ['a', NULL])`	`a,`	`a`	FE appended the separator while iterating array elements even when later elements were `NULL`; BE skips `NULL` arguments and only inserts separators between retained values.
`RIGHT('😀a', 1)`	`😀`	`a`	FE mixed Java UTF-16 indexes with SQL character positions after counting code points, so the substring start was computed from incompatible units.
`INSTR('😀a', 'a')`	`3`	`2`	FE returned `String.indexOf(...) + 1`, which is a UTF-16 char offset; BE returns the 1-based character position.
`UPPER('éßi')`	`ÉßI`	`ÉSSI`	FE used per-`charCharacter.toUpperCase`, which cannot perform Unicode string mappings such as `ß -> SS`; BE uses ICU string case conversion.
`LOWER('ÉİA')`	`éia`	`éi̇a` (`U+00E9 U+0069 U+0307 U+0061`)	FE used per-`char` lowercasing and lost the combining dot from `İ`; BE/ICU lowercases `İ` to `i + U+0307`.
`INITCAP('ßETA İSTANBUL')`	`ßeta İstanbul`	`ßeta İStanbul` (`İ` lowercases to `i + U+0307`, then the combining mark starts a new word for `S`)	FE did per-`char` case conversion directly on the original string; BE lowercases the whole string first with ICU, then uppercases the first alphanumeric code point after each non-alphanumeric code point.
`STRCMP('😀','')`	`-1`	`1`	FE used Java UTF-16 lexical ordering; BE compares string bytes, so the UTF-8 byte sequence for `😀` sorts after ``.
`FIND_IN_SET('', 'a,')`	`2`	`0`	FE used Java `split(',', -1)`, which preserves the trailing empty token; BE's scan loop stops when the next token would start at the string length, so the trailing empty token is not matched.
`PARSE_URL('http://h/p%20x?q=a+b%20c&k=v#r', 'PATH')`	`/p x`	`/p%20x`	FE used `java.net.URI`, which decodes escaped bytes; BE `UrlParser` returns raw URL substrings.
`PARSE_URL('http://h/p%20x?q=a+b%20c&k=v#r', 'QUERY')`	`q=a+b c&k=v`	`q=a+b%20c&k=v`	Same raw-substring vs decoded-URI mismatch.
`EXTRACT_URL_PARAMETER('http://h/p%20x?q=a+b%20c&k=v#r', 'q')`	`a+b c`	`a+b%20c`	FE extracted from the decoded URI query; BE extracts the raw parameter value from the original query string.
`MAKETIME(1, 2, 3.9999995)`	`01:02:03.000000`	`01:02:04.000000`	FE truncated the integer seconds and then took rounded microseconds modulo `1000000`, losing the carry into seconds; BE rounds total seconds to microsecond precision before constructing the time value.
`MAKETIME(1, 2, 59.9999995)`	`01:02:59.000000`	`01:03:00.000000`	Same lost microsecond carry, this time across the minute boundary.
`FIELD(CAST('-0.0' AS DOUBLE), CAST('0.0' AS DOUBLE), CAST('-0.0' AS DOUBLE))`	`2`	`1`	FE used boxed floating equality semantics that distinguish `-0.0` from `0.0`; BE compares floating values numerically, so `-0.0 == 0.0` and the first candidate matches.

The PR also makes MD5/MD5SUM folding explicitly use UTF-8 bytes, matching BE and avoiding JVM-default-charset dependent folding results.

Release note

None

Check List (For Author)

Test: Unit Test
- mvn checkstyle:check -pl fe-core
- mvn -pl fe-core -am -DskipTests compile
Behavior changed: No
Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: FE constant folding for string functions had two inconsistencies with backend execution. The array form of concat_ws appended separators while scanning every non-final element, so arrays ending in NULL could keep a trailing separator after folding. The literal vararg form with NULL was not folded with the same skip-NULL semantics as BE. The MD5 folding path also used the JVM default charset when converting Java strings to bytes, while BE hashes the UTF-8 bytes stored in string columns. This change joins only non-NULL concat_ws inputs with separators between retained values, falls back to BE for unsupported array varargs, and uses UTF-8 bytes for MD5 folding. ### Release note None ### Check List (For Author) - Test: Regression test / Manual test - mvn checkstyle:check -pl fe-core - mvn -pl fe-core -am -DskipTests compile - ./run-regression-test.sh --run -d query_p0/sql_functions/string_functions -s string_functions_all (failed against the already-running old FE, reproducing the pre-fix concat_ws mismatch: folded a, vs BE a) - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Constant folding for several FE executable functions produced results different from BE execution. String functions mixed Java UTF-16 indexes with SQL character positions, Unicode case conversion used per-char Java helpers instead of locale-independent string conversion or BE-like single-code-point initcap, strcmp used UTF-16 ordering instead of UTF-8 byte ordering, URL helpers decoded URI components instead of returning raw parser substrings, maketime dropped microsecond rounding carry, and floating field comparison distinguished -0.0 from 0.0. This change aligns those FE folding paths with BE behavior and adds testFoldConst coverage for the mismatched expressions. ### Release note None ### Check List (For Author) - Test: Unit Test - mvn checkstyle:check -pl fe-core - mvn -pl fe-core -am -DskipTests compile - Behavior changed: No - Does this need documentation: No

hello-stephen · 2026-06-26T06:31:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Issue Number: N/A Related PR: N/A Problem Summary: FE constant folding for find_in_set used Java split with trailing empty tokens, but BE does not match an empty token after the final comma because its scan loop stops when the next token starts at the string length. FE also made the floating field comparison clearer by explicitly comparing primitive float and double values, preserving BE numeric equality for -0.0 and 0.0. None - Test: Unit Test - mvn checkstyle:check -pl fe-core - mvn -pl fe-core -am -DskipTests compile - Behavior changed: No - Does this need documentation: No

morrySnow · 2026-06-26T06:46:45Z

/review

morrySnow · 2026-06-26T06:48:17Z

run buildall

github-actions

Automated review completed. I found no blocking issues and am leaving this as a summary-only review.

Critical checkpoints:

Goal/test proof: The PR aligns FE executable constant folding for the touched string/time functions with BE behavior. I checked the FE implementations against BE MakeTimeImpl, FunctionStringConcatWs, StringInStrImpl, FunctionRight, ICU case conversion, FunctionStrcmp, FindInSetOp, and UrlParser; the added testFoldConst cases compare folded vs non-folded execution.
Scope/parallel paths: The change is focused on FE constant-folding methods and matching regression calls. I did not find a missing parallel path in the reviewed scope.
Concurrency/lifecycle/config/protocol/data writes: Not applicable; no concurrency, persistent state, config item, FE-BE protocol, storage format, transaction, or data-write behavior changed.
Tests/validation: Added regression testFoldConst coverage for the fixed mismatches. I ran git diff --check on the GitHub-scoped changed files. I could not run FE/regression tests in this runner because thirdparty/installed / thirdparty/installed/bin/protoc is absent.
User focus: No additional user-provided review focus was supplied.

Subagent conclusions:

optimizer-rewrite: no candidates in first pass; convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.
tests-session-config: no candidates in first pass; convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.
No inline comments were proposed or submitted.

hello-stephen · 2026-06-26T07:54:55Z

TPC-H: Total hot run time: 29544 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed5f4c6e5561037d7e8d1ed05822236ef333f1d0, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17836	4192	4057	4057
q2	2035	326	189	189
q3	10281	1452	847	847
q4	4686	477	345	345
q5	7490	872	586	586
q6	191	180	138	138
q7	812	844	646	646
q8	9444	1656	1769	1656
q9	5640	4531	4517	4517
q10	6844	1785	1514	1514
q11	446	279	244	244
q12	635	429	291	291
q13	18105	3328	2735	2735
q14	262	272	246	246
q15	q16	787	780	711	711
q17	917	876	871	871
q18	6977	5839	5885	5839
q19	1298	1296	1081	1081
q20	482	394	261	261
q21	6247	2579	2464	2464
q22	442	361	306	306
Total cold run time: 101857 ms
Total hot run time: 29544 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4474	4410	4422	4410
q2	318	344	226	226
q3	4594	4978	4369	4369
q4	2101	2172	1373	1373
q5	4437	4341	4322	4322
q6	244	183	132	132
q7	1710	1999	1744	1744
q8	2790	2187	2195	2187
q9	8227	8492	8195	8195
q10	4845	4751	4230	4230
q11	560	424	394	394
q12	749	809	549	549
q13	3270	3659	2963	2963
q14	307	296	285	285
q15	q16	720	739	631	631
q17	1385	1354	1488	1354
q18	7893	7279	7219	7219
q19	1162	1101	1109	1101
q20	2212	2197	1970	1970
q21	5314	4655	4438	4438
q22	528	480	404	404
Total cold run time: 57840 ms
Total hot run time: 52496 ms

hello-stephen · 2026-06-26T08:05:50Z

TPC-DS: Total hot run time: 171880 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed5f4c6e5561037d7e8d1ed05822236ef333f1d0, data reload: false
query5	4347	635	504	504
query6	446	199	185	185
query7	4815	568	311	311
query8	341	193	171	171
query9	8789	4016	4041	4016
query10	444	315	269	269
query11	5909	2317	2144	2144
query12	155	102	101	101
query13	1267	603	438	438
query14	6393	5307	4937	4937
query14_1	4276	4288	4289	4288
query15	218	204	187	187
query16	1056	510	455	455
query17	1156	723	596	596
query18	2718	458	343	343
query19	201	178	138	138
query20	116	105	108	105
query21	221	135	112	112
query22	13720	13607	13314	13314
query23	17582	16522	16144	16144
query23_1	16267	16286	16278	16278
query24	7725	1766	1274	1274
query24_1	1301	1305	1294	1294
query25	535	433	360	360
query26	1305	309	162	162
query27	2670	577	337	337
query28	4439	2020	1978	1978
query29	1059	587	494	494
query30	305	232	211	211
query31	1105	1070	946	946
query32	113	60	59	59
query33	508	321	247	247
query34	1185	1143	642	642
query35	754	787	656	656
query36	1380	1379	1187	1187
query37	148	99	90	90
query38	1895	1710	1657	1657
query39	922	921	886	886
query39_1	915	880	869	869
query40	213	124	101	101
query41	68	62	62	62
query42	90	87	93	87
query43	315	322	275	275
query44	1410	776	779	776
query45	204	189	180	180
query46	1084	1218	761	761
query47	2390	2397	2282	2282
query48	392	399	294	294
query49	581	449	323	323
query50	1018	373	266	266
query51	4442	4397	4338	4338
query52	81	82	71	71
query53	261	259	191	191
query54	275	233	221	221
query55	75	72	68	68
query56	253	230	225	225
query57	1438	1404	1335	1335
query58	251	232	218	218
query59	1578	1597	1464	1464
query60	296	261	241	241
query61	174	175	176	175
query62	692	664	551	551
query63	234	191	201	191
query64	2562	818	662	662
query65	4866	4803	4749	4749
query66	1777	475	362	362
query67	28894	28754	28660	28660
query68	3146	1673	836	836
query69	430	308	265	265
query70	1079	974	937	937
query71	288	233	214	214
query72	2822	2646	2341	2341
query73	842	767	417	417
query74	5106	4963	4766	4766
query75	2574	2539	2177	2177
query76	2298	1203	775	775
query77	350	391	312	312
query78	12495	12435	11930	11930
query79	1449	1187	740	740
query80	1271	483	394	394
query81	514	282	236	236
query82	610	159	119	119
query83	323	272	253	253
query84	262	143	111	111
query85	898	531	409	409
query86	421	283	268	268
query87	1836	1848	1779	1779
query88	3688	2785	2751	2751
query89	421	393	343	343
query90	1942	184	178	178
query91	171	163	134	134
query92	65	63	57	57
query93	1588	1458	913	913
query94	705	338	302	302
query95	694	380	337	337
query96	1038	824	377	377
query97	2713	2677	2649	2649
query98	233	212	207	207
query99	1192	1163	1038	1038
Total cold run time: 258514 ms
Total hot run time: 171880 ms

hello-stephen · 2026-06-26T08:10:46Z

ClickBench: Total hot run time: 25.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed5f4c6e5561037d7e8d1ed05822236ef333f1d0, data reload: false
query1	0.00	0.01	0.00
query2	0.10	0.05	0.05
query3	0.25	0.14	0.16
query4	1.61	0.14	0.13
query5	0.25	0.22	0.22
query6	1.26	1.09	1.06
query7	0.04	0.00	0.00
query8	0.06	0.04	0.03
query9	0.40	0.31	0.32
query10	0.54	0.54	0.53
query11	0.18	0.15	0.14
query12	0.19	0.16	0.14
query13	0.47	0.48	0.46
query14	1.03	1.01	1.00
query15	0.60	0.60	0.60
query16	0.32	0.32	0.30
query17	1.13	1.12	1.11
query18	0.22	0.21	0.21
query19	2.03	1.98	1.95
query20	0.02	0.01	0.02
query21	15.43	0.23	0.13
query22	4.92	0.05	0.06
query23	16.08	0.31	0.12
query24	2.90	0.43	0.32
query25	0.11	0.07	0.05
query26	0.73	0.21	0.17
query27	0.05	0.04	0.03
query28	3.54	0.89	0.56
query29	12.47	4.32	3.46
query30	0.27	0.15	0.16
query31	2.77	0.59	0.30
query32	3.22	0.60	0.49
query33	3.18	3.22	3.20
query34	15.74	4.15	3.47
query35	3.48	3.52	3.53
query36	0.57	0.43	0.43
query37	0.09	0.07	0.06
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.16
query41	0.07	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.66 s
Total hot run time: 25.21 s

…rd with BE Reviewed the merged levenshtein/hamming_distance (apache#60412) and damerau_levenshtein_distance (apache#65278, apache#66236) PRs for consistency with this one. Found and fixed a real FE/BE divergence in the process: - The FE constant-fold length guard checked Java code point count, while the BE guard checks UTF-8 byte length. For multi-byte input the two disagree (e.g. ~30000 3-byte characters is under the FE's 65535 code-point cap but over BE's 65535-byte cap), so a literal expression could fold successfully on FE while the same value would be rejected by BE if read from a column. FE now measures UTF-8 bytes too, matching function_string_similarity.cpp exactly (mirrors the fix pattern in apache#64881, "Align constant folding with BE results"). - Added regression coverage for astral-plane (surrogate-pair/4-byte UTF-8) characters on all three functions, matching the precedent set for RIGHT/INSTR in apache#64881.

morrySnow added 2 commits June 26, 2026 13:55

morrySnow changed the title ~~[fix](fe) Align constant folding with BE results~~[fix](function) Align constant folding with BE resultsJun 26, 2026

morrySnow force-pushed the fix-constant-folding branch from e7bd52f to ed5f4c6Compare June 26, 2026 06:46

morrySnow added dev/4.1.x dev/3.1.x dev/4.0.x and removed dev/3.1.x labels Jun 26, 2026

morrySnow marked this pull request as ready for review June 26, 2026 06:48

morrySnow requested review from 924060929, englefly and starocean999 as code owners June 26, 2026 06:48

github-actionsBot reviewed Jun 26, 2026
View reviewed changes

starocean999 approved these changes Jun 26, 2026
View reviewed changes

morrySnow merged commit 8dca6c8 into apache:masterJun 26, 2026
36 of 37 checks passed

morrySnow deleted the fix-constant-folding branch June 26, 2026 09:45

github-actionsBot added dev/4.0.x-conflict dev/4.1.x-conflict labels Jun 26, 2026

Navigation Menu

Uh oh!

[fix](function) Align constant folding with BE results - #64881

[fix](function) Align constant folding with BE results#64881
morrySnow merged 3 commits into
apache:masterfrom
morrySnow:fix-constant-folding

morrySnow commented Jun 26, 2026•
edited
Loading

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

morrySnow commented Jun 26, 2026

Uh oh!

morrySnow commented Jun 26, 2026

Uh oh!

github-actionsBot left a comment

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

morrySnow commented Jun 26, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Mismatched expressions

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

morrySnow commented Jun 26, 2026

Uh oh!

morrySnow commented Jun 26, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

hello-stephen commented Jun 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

morrySnow commented Jun 26, 2026•
edited
Loading