[fix](load) Load column mapping with expressions by 0AyanamiRei · Pull Request #58353 · apache/doris

0AyanamiRei · 2025-11-25T08:00:42Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

before this pr: we can only use c1 = 72105107105, c2 = "ykiko", these constant values in load.
after this pr : we can use c1 = uuid(), c2 = abs(-2)+1, c3 = now(), these constant function.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-11-25T08:00:58Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

0AyanamiRei · 2025-11-26T03:04:23Z

run buildall

doris-robot · 2025-11-26T03:43:13Z

TPC-H: Total hot run time: 34029 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06790e0ca49d380fe26523d38185fd8b06b6e3f2, data reload: false
------ Round 1 ----------------------------------
q1	17631	5007	4917	4917
q2	2005	327	213	213
q3	10257	1306	708	708
q4	10216	857	351	351
q5	7543	2427	2324	2324
q6	183	168	135	135
q7	941	798	649	649
q8	9385	1366	1025	1025
q9	7020	5286	5353	5286
q10	6901	2251	1823	1823
q11	513	313	300	300
q12	345	359	217	217
q13	17826	3658	3077	3077
q14	232	247	220	220
q15	603	512	512	512
q16	1032	1038	953	953
q17	592	863	346	346
q18	7442	7077	6984	6984
q19	1164	947	523	523
q20	345	332	230	230
q21	3727	3138	2279	2279
q22	1031	1024	957	957
Total cold run time: 106934 ms
Total hot run time: 34029 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5018	4914	4948	4914
q2	313	400	307	307
q3	2139	2624	2329	2329
q4	1331	1794	1324	1324
q5	4240	4395	4479	4395
q6	210	171	128	128
q7	2058	1942	1818	1818
q8	2615	2735	2662	2662
q9	7507	7528	7585	7528
q10	3106	3231	2812	2812
q11	605	519	492	492
q12	684	749	616	616
q13	3718	3807	3315	3315
q14	310	303	297	297
q15	564	508	510	508
q16	1094	1140	1080	1080
q17	1184	1563	1507	1507
q18	7923	7899	7496	7496
q19	809	813	787	787
q20	2053	2152	1942	1942
q21	4832	4311	4224	4224
q22	1085	1060	1039	1039
Total cold run time: 53398 ms
Total hot run time: 51520 ms

doris-robot · 2025-11-26T03:54:43Z

TPC-DS: Total hot run time: 184768 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06790e0ca49d380fe26523d38185fd8b06b6e3f2, data reload: false
query1	1051	407	396	396
query2	6584	1616	1570	1570
query3	6764	228	223	223
query4	25323	22922	22642	22642
query5	4406	607	461	461
query6	328	251	230	230
query7	4659	497	302	302
query8	311	289	244	244
query9	8720	2619	2631	2619
query10	533	380	312	312
query11	15753	14739	14619	14619
query12	193	120	114	114
query13	1694	582	457	457
query14	10739	8893	8855	8855
query15	234	200	195	195
query16	7334	704	534	534
query17	1256	768	658	658
query18	2006	458	346	346
query19	214	206	186	186
query20	130	129	123	123
query21	225	139	116	116
query22	3852	3927	3830	3830
query23	32765	32032	32106	32032
query24	8559	2443	2408	2408
query25	643	556	474	474
query26	1242	285	165	165
query27	2745	509	362	362
query28	4377	2185	2155	2155
query29	860	643	521	521
query30	310	240	211	211
query31	870	709	640	640
query32	87	79	73	73
query33	613	387	335	335
query34	810	880	574	574
query35	797	834	731	731
query36	931	958	854	854
query37	127	106	90	90
query38	3384	3353	3264	3264
query39	1485	1437	1398	1398
query40	227	128	120	120
query41	65	62	61	61
query42	126	116	114	114
query43	460	453	445	445
query44	1248	781	779	779
query45	199	189	177	177
query46	892	1006	641	641
query47	1709	1736	1639	1639
query48	403	430	349	349
query49	790	489	420	420
query50	656	683	420	420
query51	3930	3979	3925	3925
query52	114	112	109	109
query53	236	253	197	197
query54	312	294	274	274
query55	88	85	83	83
query56	321	341	321	321
query57	1157	1185	1105	1105
query58	283	274	276	274
query59	2425	2475	2396	2396
query60	362	350	343	343
query61	166	163	160	160
query62	782	721	658	658
query63	233	193	190	190
query64	4604	1174	881	881
query65	4059	3976	3978	3976
query66	1184	430	366	366
query67	15260	15382	15010	15010
query68	8513	981	648	648
query69	523	336	299	299
query70	1302	1244	1188	1188
query71	472	344	320	320
query72	5621	4917	4861	4861
query73	730	574	377	377
query74	8613	8879	8335	8335
query75	4006	3320	2860	2860
query76	3745	1148	721	721
query77	825	398	304	304
query78	9451	9796	8967	8967
query79	1290	850	611	611
query80	710	590	507	507
query81	489	272	257	257
query82	280	177	134	134
query83	303	275	259	259
query84	304	113	99	99
query85	885	477	436	436
query86	340	327	275	275
query87	3528	3524	3490	3490
query88	2940	2287	2267	2267
query89	391	329	298	298
query90	2077	229	222	222
query91	174	172	147	147
query92	87	72	64	64
query93	1179	1012	678	678
query94	737	451	352	352
query95	502	412	398	398
query96	507	587	283	283
query97	2907	2994	2885	2885
query98	243	218	212	212
query99	1398	1380	1262	1262
Total cold run time: 270277 ms
Total hot run time: 184768 ms

doris-robot · 2025-11-26T03:59:51Z

ClickBench: Total hot run time: 27.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06790e0ca49d380fe26523d38185fd8b06b6e3f2, data reload: false
query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.24
query6	1.17	0.66	0.65
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.51	0.51
query10	0.58	0.58	0.57
query11	0.16	0.11	0.12
query12	0.16	0.12	0.12
query13	0.62	0.61	0.60
query14	1.02	1.00	0.99
query15	0.86	0.84	0.85
query16	0.40	0.39	0.39
query17	1.01	1.05	1.00
query18	0.21	0.22	0.20
query19	1.94	1.77	1.82
query20	0.01	0.01	0.01
query21	15.44	0.20	0.13
query22	5.10	0.07	0.05
query23	15.69	0.26	0.10
query24	3.21	0.58	0.68
query25	0.07	0.07	0.05
query26	0.13	0.15	0.13
query27	0.06	0.06	0.06
query28	4.91	1.16	0.95
query29	12.57	3.95	3.20
query30	0.27	0.14	0.11
query31	2.82	0.59	0.40
query32	3.23	0.54	0.47
query33	3.11	3.03	3.03
query34	15.93	5.19	4.59
query35	4.55	4.53	4.53
query36	0.67	0.49	0.48
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 99.4 s
Total hot run time: 27.58 s

hello-stephen · 2025-11-26T04:13:33Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-11-26T06:05:23Z

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

0AyanamiRei · 2025-11-26T07:48:48Z

run buildall

hello-stephen · 2025-11-26T08:49:25Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-11-26T08:58:59Z

TPC-H: Total hot run time: 34497 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 631123d7a4cd4d5abe7b56963ae0e23c4ef48855, data reload: false
------ Round 1 ----------------------------------
q1	17633	5117	4977	4977
q2	2056	311	206	206
q3	10249	1303	730	730
q4	10228	949	378	378
q5	7528	2473	2316	2316
q6	194	176	143	143
q7	932	790	629	629
q8	9355	1417	1086	1086
q9	7133	5327	5377	5327
q10	6922	2279	1849	1849
q11	507	305	279	279
q12	363	368	225	225
q13	17800	3707	3108	3108
q14	230	235	213	213
q15	595	513	518	513
q16	1043	1027	957	957
q17	595	865	367	367
q18	7621	7273	7078	7078
q19	1103	1098	562	562
q20	356	345	227	227
q21	3870	3245	2373	2373
q22	1033	1016	954	954
Total cold run time: 107346 ms
Total hot run time: 34497 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5006	4965	4957	4957
q2	342	408	334	334
q3	2203	2664	2318	2318
q4	1345	1813	1344	1344
q5	4289	4603	4624	4603
q6	232	185	138	138
q7	2112	1959	1843	1843
q8	2683	2670	2542	2542
q9	7626	7575	7533	7533
q10	3141	3246	2857	2857
q11	605	539	508	508
q12	679	790	648	648
q13	3739	3870	3367	3367
q14	291	306	266	266
q15	547	534	509	509
q16	1170	1154	1091	1091
q17	1233	1576	1355	1355
q18	7881	7780	7529	7529
q19	807	986	1121	986
q20	2103	2046	1977	1977
q21	5045	4387	4351	4351
q22	1090	1030	1016	1016
Total cold run time: 54169 ms
Total hot run time: 52072 ms

doris-robot · 2025-11-26T09:10:42Z

TPC-DS: Total hot run time: 184544 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 631123d7a4cd4d5abe7b56963ae0e23c4ef48855, data reload: false
query1	1037	457	386	386
query2	6565	1599	1615	1599
query3	6758	223	223	223
query4	26065	22703	22317	22317
query5	4405	637	488	488
query6	335	250	241	241
query7	4648	495	294	294
query8	302	255	244	244
query9	8690	2567	2547	2547
query10	495	354	307	307
query11	15530	14892	14725	14725
query12	187	116	115	115
query13	1690	578	459	459
query14	11013	8956	8973	8956
query15	212	202	183	183
query16	7380	702	551	551
query17	1258	778	657	657
query18	2007	443	343	343
query19	219	211	187	187
query20	135	128	137	128
query21	220	139	118	118
query22	3841	4010	3780	3780
query23	33210	32149	31971	31971
query24	8480	2395	2799	2395
query25	652	537	492	492
query26	1237	282	171	171
query27	2731	497	345	345
query28	4366	2147	2122	2122
query29	853	647	526	526
query30	310	245	214	214
query31	878	722	645	645
query32	84	76	78	76
query33	614	399	336	336
query34	847	883	528	528
query35	819	834	779	779
query36	919	943	865	865
query37	121	107	92	92
query38	3355	3398	3377	3377
query39	1486	1462	1424	1424
query40	231	126	121	121
query41	65	62	77	62
query42	130	112	111	111
query43	441	476	423	423
query44	1224	754	745	745
query45	205	189	191	189
query46	894	998	648	648
query47	1713	1723	1687	1687
query48	405	423	333	333
query49	785	493	409	409
query50	652	692	409	409
query51	3908	3946	3952	3946
query52	109	111	99	99
query53	242	273	197	197
query54	323	316	279	279
query55	95	93	91	91
query56	336	324	312	312
query57	1183	1174	1142	1142
query58	288	275	269	269
query59	2396	2585	2397	2397
query60	383	352	344	344
query61	170	158	163	158
query62	787	720	647	647
query63	224	193	193	193
query64	4584	1195	885	885
query65	4097	3970	3996	3970
query66	1154	440	328	328
query67	15321	14970	14970	14970
query68	8465	918	630	630
query69	505	342	302	302
query70	1316	1273	1257	1257
query71	488	337	314	314
query72	5765	4977	5009	4977
query73	680	574	343	343
query74	8920	8687	8501	8501
query75	4047	3344	2808	2808
query76	3767	1132	711	711
query77	811	407	305	305
query78	9448	9716	8863	8863
query79	2336	795	587	587
query80	674	565	482	482
query81	504	283	242	242
query82	457	162	130	130
query83	312	271	255	255
query84	309	113	102	102
query85	925	484	491	484
query86	346	293	281	281
query87	3569	3550	3359	3359
query88	3454	2284	2314	2284
query89	385	326	294	294
query90	1976	233	230	230
query91	177	176	148	148
query92	89	70	64	64
query93	1546	975	672	672
query94	735	432	341	341
query95	507	406	399	399
query96	503	574	279	279
query97	2944	2999	2826	2826
query98	243	213	206	206
query99	1433	1380	1298	1298
Total cold run time: 273935 ms
Total hot run time: 184544 ms

doris-robot · 2025-11-26T09:15:48Z

ClickBench: Total hot run time: 27.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 631123d7a4cd4d5abe7b56963ae0e23c4ef48855, data reload: false
query1	0.05	0.05	0.05
query2	0.09	0.05	0.04
query3	0.26	0.08	0.08
query4	1.60	0.11	0.12
query5	0.26	0.25	0.26
query6	1.21	0.64	0.64
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.58	0.52	0.52
query10	0.58	0.57	0.58
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	1.00	1.01	1.01
query15	0.84	0.84	0.83
query16	0.40	0.38	0.39
query17	1.06	1.00	1.05
query18	0.21	0.19	0.20
query19	1.90	1.81	1.82
query20	0.02	0.01	0.01
query21	15.44	0.19	0.14
query22	5.03	0.07	0.04
query23	15.71	0.25	0.10
query24	2.23	0.56	0.87
query25	0.08	0.07	0.06
query26	0.15	0.13	0.13
query27	0.07	0.05	0.06
query28	5.02	1.13	0.95
query29	12.56	3.90	3.25
query30	0.28	0.15	0.12
query31	2.82	0.60	0.39
query32	3.23	0.56	0.46
query33	3.06	3.06	3.14
query34	15.83	5.10	4.52
query35	4.50	4.63	4.54
query36	0.68	0.50	0.49
query37	0.10	0.07	0.07
query38	0.06	0.04	0.03
query39	0.04	0.03	0.03
query40	0.19	0.15	0.13
query41	0.09	0.03	0.02
query42	0.03	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.31 s
Total hot run time: 27.58 s

hello-stephen · 2025-11-26T11:38:42Z

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

liaoxin01

LGTM

github-actions · 2025-11-26T13:10:16Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-26T13:10:19Z

PR approved by anyone and no changes requested.

sollhui

LGTM

### What problem does this PR solve? Related PR: [#xxx](#56041) Problem Summary: improve #56041 before this pr: we can only use `c1 = 72105107105`, `c2 = "ykiko"`, these constant values in load. after this pr : we can use `c1 = uuid()`, `c2 = abs(-2)+1`, `c3 = now()`, these constant function.

…58437) Cherry-picked from #58353 Co-authored-by: Refrain <113875799+0AyanamiRei@users.noreply.github.com>

### What problem does this PR solve? Related PR: [#xxx](apache#56041) Problem Summary: improve apache#56041 before this pr: we can only use `c1 = 72105107105`, `c2 = "ykiko"`, these constant values in load. after this pr : we can use `c1 = uuid()`, `c2 = abs(-2)+1`, `c3 = now()`, these constant function.

…ressions (#59149) Related PR: #58353 #56041

0AyanamiRei added 2 commits November 25, 2025 09:53

FIX
cfec228

f
cc59198

0AyanamiRei changed the title ([fix](load) Load column mapping with expressionsNov 25, 2025

0AyanamiRei added 2 commits November 25, 2025 16:38

f
4970fcc

f
06790e0

add test
631123d

liaoxin01 approved these changes Nov 26, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 26, 2025

github-actionsBot added the reviewed label Nov 26, 2025

liaoxin01 added dev/3.1.x dev/4.0.x labels Nov 27, 2025

sollhui approved these changes Nov 27, 2025
View reviewed changes

liaoxin01 merged commit 645444a into apache:masterNov 27, 2025
30 of 31 checks passed

github-actionsBot added the dev/3.1.x-conflict label Nov 27, 2025

github-actionsBot mentioned this pull request Nov 27, 2025
branch-4.0: [fix](load) Load column mapping with expressions #58353 #58437
Merged

yiguolei pushed a commit that referenced this pull request Nov 28, 2025
branch-4.0: [fix](load) Load column mapping with expressions#58353(#…
807a625
…58437) Cherry-picked from #58353 Co-authored-by: Refrain <113875799+0AyanamiRei@users.noreply.github.com>

yiguolei added dev/4.0.2-merged and removed dev/4.0.x labels Nov 28, 2025

morrySnow removed dev/3.1.x dev/3.1.x-conflict labels Dec 4, 2025

0AyanamiRei mentioned this pull request Dec 18, 2025
[fix](load) Fix the sensitivity issue of load column mapping with expressions #59149
Merged
16 tasks

dataroaring pushed a commit that referenced this pull request Dec 18, 2025
[fix](load) Fix the sensitivity issue of load column mapping with exp…
6d98eab
…ressions (#59149) Related PR: #58353 #56041

github-actionsBot pushed a commit that referenced this pull request Dec 18, 2025
[fix](load) Fix the sensitivity issue of load column mapping with exp…
296ffd9
…ressions (#59149) Related PR: #58353 #56041

Uh oh!

Conversation

0AyanamiRei commented Nov 25, 2025• edited by liaoxin01 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Nov 25, 2025

Uh oh!

0AyanamiRei commented Nov 26, 2025

Uh oh!

doris-robot commented Nov 26, 2025

Uh oh!

doris-robot commented Nov 26, 2025

Uh oh!

doris-robot commented Nov 26, 2025

Uh oh!

hello-stephen commented Nov 26, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Nov 26, 2025

FE Regression Coverage Report

Uh oh!

0AyanamiRei commented Nov 26, 2025

Uh oh!

hello-stephen commented Nov 26, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Nov 26, 2025

Uh oh!

doris-robot commented Nov 26, 2025

Uh oh!

doris-robot commented Nov 26, 2025

Uh oh!

hello-stephen commented Nov 26, 2025

FE Regression Coverage Report

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 26, 2025

Uh oh!

github-actionsBot commented Nov 26, 2025

Uh oh!

sollhui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

0AyanamiRei commented Nov 25, 2025•
edited by liaoxin01
Loading