[fix](fe) Preserve path partition columns with CSV schema by 0AyanamiRei · Pull Request #67407 · apache/doris

0AyanamiRei · 2026-09-01T23:52:39Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: File table-valued functions returned only explicitly declared CSV columns when csv_schema was set, so columns derived from path_partition_keys disappeared from the table schema. Append non-conflicting path partition columns to the explicit CSV schema in their declared order and reject case-insensitive conflicts before they can create a duplicate schema.

Release note

CSV file table-valued functions now expose path partition columns when csv_schema is specified and reject conflicting column names.

Check List (For Author)

Test: FE unit test and regression test
- FE unit test: ExternalFileTableValuedFunctionTest (5 tests)
- Regression test: test_csv_schema_with_path_partition_keys, including conflict rejection
Behavior changed: Yes. Path partition columns remain available with an explicit CSV schema.
Does this need documentation: No

hello-stephen · 2026-09-01T23:52:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

0AyanamiRei · 2026-09-02T00:53:00Z

run buildall

hello-stephen · 2026-09-02T02:14:35Z

TPC-H: Total hot run time: 16828 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 465e988df0ad58b2d652b3a56c94542759bce9b1, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17587	3035	3032	3032
q2	2101	259	217	217
q3	10226	847	526	526
q4	4674	246	199	199
q5	7680	563	385	385
q6	135	113	93	93
q7	527	505	389	389
q8	9242	951	913	913
q9	3496	2352	2356	2352
q10	6510	1022	709	709
q11	400	201	182	182
q12	610	257	202	202
q13	18119	1559	1162	1162
q14	165	151	144	144
q15	q16	434	393	367	367
q17	1394	885	770	770
q18	3115	2226	2227	2226
q19	1265	866	742	742
q20	369	288	208	208
q21	5585	1774	1810	1774
q22	337	264	236	236
Total cold run time: 93971 ms
Total hot run time: 16828 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3395	3343	3341	3341
q2	515	391	368	368
q3	2221	2788	2198	2198
q4	1179	1158	884	884
q5	2150	2110	2097	2097
q6	169	126	85	85
q7	1047	932	853	853
q8	1597	1407	1420	1407
q9	3147	3136	3089	3089
q10	1886	1786	1618	1618
q11	353	266	250	250
q12	451	428	337	337
q13	1476	1536	1135	1135
q14	176	168	168	168
q15	q16	396	397	363	363
q17	3537	3285	3193	3193
q18	4762	4385	4715	4385
q19	874	899	886	886
q20	986	950	834	834
q21	3720	3004	3262	3004
q22	393	353	318	318
Total cold run time: 34430 ms
Total hot run time: 30813 ms

hello-stephen · 2026-09-02T02:21:01Z

TPC-DS: Total hot run time: 81714 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 465e988df0ad58b2d652b3a56c94542759bce9b1, data reload: false
query5	4250	411	334	334
query6	377	144	126	126
query7	4948	405	241	241
query8	292	135	114	114
query9	8677	2910	2892	2892
query10	383	219	177	177
query11	5365	1058	913	913
query12	120	70	67	67
query13	1190	431	332	332
query14	6029	2189	2097	2097
query14_1	1976	1946	1955	1946
query15	179	124	111	111
query16	922	363	357	357
query17	810	442	369	369
query18	2347	315	247	247
query19	164	138	107	107
query20	76	70	69	69
query21	202	101	86	86
query22	5389	5224	5251	5224
query23	6646	6263	6068	6068
query23_1	6001	6089	5798	5798
query24	7337	1144	801	801
query24_1	771	782	787	782
query25	404	293	238	238
query26	1232	233	127	127
query27	2800	397	251	251
query28	4706	1492	1506	1492
query29	930	416	345	345
query30	248	157	129	129
query31	823	399	322	322
query32	128	76	68	68
query33	455	206	166	166
query34	972	801	479	479
query35	403	397	334	334
query36	554	614	523	523
query37	124	80	66	66
query38	996	830	832	830
query39	494	480	487	480
query39_1	471	450	444	444
query40	204	93	72	72
query41	53	54	51	51
query42	74	75	74	74
query43	251	241	213	213
query44	1026	530	545	530
query45	110	101	99	99
query46	766	855	533	533
query47	770	761	702	702
query48	314	306	231	231
query49	554	240	194	194
query50	780	264	199	199
query51	8024	8036	8069	8036
query52	72	68	62	62
query53	196	196	153	153
query54	218	189	158	158
query55	75	57	58	57
query56	189	157	151	151
query57	691	713	614	614
query58	193	151	160	151
query59	1204	1239	1084	1084
query60	231	182	173	173
query61	122	112	134	112
query62	389	205	178	178
query63	165	141	142	141
query64	2856	680	602	602
query65	1576	1608	1588	1588
query66	1930	269	226	226
query67	9994	9658	9769	9658
query68	2752	1147	758	758
query69	341	222	190	190
query70	665	607	606	606
query71	239	175	162	162
query72	2307	1858	1559	1559
query73	669	602	344	344
query74	1586	1232	1128	1128
query75	1149	1080	949	949
query76	2283	734	556	556
query77	260	260	209	209
query78	3873	3649	3289	3289
query79	1199	814	583	583
query80	1046	331	270	270
query81	469	152	136	136
query82	593	128	98	98
query83	286	215	197	197
query84	289	113	87	87
query85	829	366	292	292
query86	353	178	173	173
query87	1012	951	882	882
query88	2744	2102	2108	2102
query89	279	195	177	177
query90	1835	127	126	126
query91	128	161	102	102
query92	78	65	74	65
query93	1152	1153	745	745
query94	553	256	222	222
query95	529	246	300	246
query96	787	572	270	270
query97	1110	1071	1003	1003
query98	147	132	140	132
query99	428	346	309	309
Total cold run time: 174644 ms
Total hot run time: 81714 ms

hello-stephen · 2026-09-02T02:25:23Z

ClickBench: Total hot run time: 14.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 465e988df0ad58b2d652b3a56c94542759bce9b1, data reload: false
query1	0.01	0.00	0.00
query2	0.07	0.03	0.03
query3	0.25	0.11	0.11
query4	1.61	0.10	0.09
query5	0.17	0.17	0.16
query6	1.22	0.69	0.70
query7	0.04	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.22
query10	0.35	0.35	0.34
query11	0.17	0.12	0.12
query12	0.15	0.12	0.12
query13	0.32	0.31	0.30
query14	0.45	0.45	0.44
query15	0.37	0.35	0.34
query16	0.21	0.23	0.22
query17	0.68	0.69	0.69
query18	0.18	0.16	0.17
query19	1.18	1.15	1.13
query20	0.02	0.01	0.01
query21	15.45	0.15	0.12
query22	5.07	0.05	0.04
query23	16.17	0.25	0.10
query24	3.01	0.33	0.25
query25	0.12	0.03	0.03
query26	0.73	0.17	0.13
query27	0.05	0.03	0.03
query28	3.62	0.57	0.29
query29	12.47	3.18	2.58
query30	0.25	0.12	0.13
query31	2.76	0.37	0.17
query32	3.51	0.32	0.23
query33	1.72	1.35	1.48
query34	15.36	2.17	1.76
query35	1.76	1.75	1.70
query36	0.46	0.30	0.29
query37	0.07	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.03
query40	0.11	0.07	0.08
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.7 s
Total hot run time: 14.61 s

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: File table-valued functions returned only explicitly declared CSV columns when csv_schema was set, so columns derived from path_partition_keys disappeared from the table schema. Append non-conflicting path partition columns to the explicit CSV schema in their declared order and reject case-insensitive conflicts before they can create a duplicate schema. ### Release note CSV file table-valued functions now expose path partition columns when csv_schema is specified and reject conflicting column names. ### Check List (For Author) - Test: FE unit test and regression test - FE unit test: ExternalFileTableValuedFunctionTest (5 tests) - Regression test: test_csv_schema_with_path_partition_keys, including conflict rejection - Behavior changed: Yes. Path partition columns remain available with an explicit CSV schema. - Does this need documentation: No

0AyanamiRei · 2026-09-03T03:45:52Z

run buildall

hello-stephen · 2026-09-03T05:23:07Z

TPC-H: Total hot run time: 16942 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b895d8a813e29dc78ef97615d66e669fbae50c77, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17563	3089	3096	3089
q2	2077	251	220	220
q3	10252	850	507	507
q4	4670	252	195	195
q5	7685	619	390	390
q6	135	115	92	92
q7	519	499	389	389
q8	9259	897	928	897
q9	3427	2369	2380	2369
q10	6530	856	721	721
q11	402	205	187	187
q12	617	252	200	200
q13	18139	1517	1154	1154
q14	153	155	139	139
q15	q16	431	393	366	366
q17	1264	917	808	808
q18	3073	2265	2230	2230
q19	1273	886	780	780
q20	371	277	203	203
q21	5633	1781	1801	1781
q22	329	267	225	225
Total cold run time: 93802 ms
Total hot run time: 16942 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3449	3348	3387	3348
q2	531	401	375	375
q3	2250	2286	2136	2136
q4	1192	1153	888	888
q5	2184	2091	2108	2091
q6	169	124	84	84
q7	1003	977	849	849
q8	1617	1419	1429	1419
q9	3130	3117	3088	3088
q10	1838	1791	1598	1598
q11	356	270	253	253
q12	458	425	345	345
q13	1504	1516	1153	1153
q14	170	170	157	157
q15	q16	401	390	363	363
q17	3637	3277	3259	3259
q18	4803	4407	4734	4407
q19	848	840	889	840
q20	1009	960	834	834
q21	3864	3126	3309	3126
q22	386	344	334	334
Total cold run time: 34799 ms
Total hot run time: 30947 ms

hello-stephen · 2026-09-03T05:23:23Z

FE UT Coverage Report

Increment line coverage 100.00% (12/12) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-03T05:29:37Z

TPC-DS: Total hot run time: 81782 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b895d8a813e29dc78ef97615d66e669fbae50c77, data reload: false
query5	4249	403	327	327
query6	396	138	125	125
query7	4948	394	235	235
query8	308	124	124	124
query9	8670	2860	2896	2860
query10	400	221	172	172
query11	5355	1027	904	904
query12	117	72	73	72
query13	1193	451	317	317
query14	6108	2216	2070	2070
query14_1	1979	1970	1955	1955
query15	173	118	110	110
query16	942	385	357	357
query17	811	469	364	364
query18	2342	336	247	247
query19	176	139	117	117
query20	76	74	79	74
query21	206	129	84	84
query22	5539	5280	5271	5271
query23	6777	6071	6075	6071
query23_1	5856	5954	6009	5954
query24	7255	1091	783	783
query24_1	764	785	771	771
query25	410	280	240	240
query26	1228	234	127	127
query27	2789	423	252	252
query28	4652	1499	1473	1473
query29	924	417	338	338
query30	248	148	128	128
query31	828	396	327	327
query32	153	74	69	69
query33	450	217	162	162
query34	997	823	476	476
query35	403	401	342	342
query36	552	550	542	542
query37	119	80	67	67
query38	1010	838	811	811
query39	492	483	442	442
query39_1	454	458	445	445
query40	201	89	75	75
query41	53	52	54	52
query42	78	70	70	70
query43	243	236	211	211
query44	1049	546	560	546
query45	107	106	100	100
query46	764	838	525	525
query47	743	775	694	694
query48	315	314	226	226
query49	532	255	180	180
query50	758	262	199	199
query51	8245	8334	8091	8091
query52	71	72	59	59
query53	190	201	147	147
query54	261	197	189	189
query55	79	67	58	58
query56	213	177	170	170
query57	679	649	706	649
query58	203	168	163	163
query59	1247	1230	1114	1114
query60	244	208	222	208
query61	140	129	145	129
query62	391	205	175	175
query63	172	142	140	140
query64	2675	674	583	583
query65	1646	1566	1616	1566
query66	1853	253	212	212
query67	9835	9691	9557	9557
query68	3003	1140	759	759
query69	357	229	196	196
query70	677	610	618	610
query71	249	176	167	167
query72	2329	1763	1606	1606
query73	656	590	362	362
query74	2013	1204	1135	1135
query75	1164	1084	943	943
query76	2382	720	534	534
query77	255	250	209	209
query78	3851	3692	3134	3134
query79	2692	848	617	617
query80	1600	352	275	275
query81	494	155	132	132
query82	1115	125	92	92
query83	272	210	200	200
query84	294	111	97	97
query85	806	381	292	292
query86	422	180	180	180
query87	1027	953	895	895
query88	2785	2110	2095	2095
query89	292	195	175	175
query90	1978	119	169	119
query91	129	123	100	100
query92	71	72	72	72
query93	1460	1174	761	761
query94	679	249	230	230
query95	526	317	233	233
query96	843	599	280	280
query97	1050	1048	1014	1014
query98	173	142	132	132
query99	449	347	316	316
Total cold run time: 178890 ms
Total hot run time: 81782 ms

hello-stephen · 2026-09-03T05:33:59Z

ClickBench: Total hot run time: 14.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b895d8a813e29dc78ef97615d66e669fbae50c77, data reload: false
query1	0.01	0.00	0.00
query2	0.08	0.03	0.04
query3	0.24	0.11	0.10
query4	1.63	0.10	0.09
query5	0.18	0.16	0.16
query6	1.27	0.68	0.67
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.28	0.22	0.21
query10	0.35	0.33	0.35
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.30	0.30	0.33
query14	0.46	0.45	0.44
query15	0.36	0.36	0.35
query16	0.23	0.23	0.21
query17	0.71	0.64	0.73
query18	0.17	0.18	0.18
query19	1.17	1.15	1.25
query20	0.01	0.01	0.01
query21	15.44	0.17	0.11
query22	5.04	0.04	0.04
query23	16.17	0.26	0.10
query24	3.01	0.28	0.24
query25	0.12	0.04	0.03
query26	0.83	0.16	0.12
query27	0.04	0.02	0.02
query28	3.62	0.57	0.28
query29	12.45	3.16	2.58
query30	0.25	0.11	0.12
query31	2.76	0.36	0.17
query32	3.54	0.33	0.24
query33	1.36	1.41	1.40
query34	15.35	2.21	1.77
query35	1.71	1.69	1.73
query36	0.51	0.29	0.28
query37	0.07	0.05	0.04
query38	0.04	0.02	0.02
query39	0.04	0.03	0.02
query40	0.12	0.08	0.07
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.47 s
Total hot run time: 14.52 s

0AyanamiRei force-pushed the fix/doris-25094-csv-schema-path-partitions branch from 465e988 to b895d8aCompare September 2, 2026 03:00

airborne12 mentioned this pull request Sep 2, 2026
branch-4.1: [fix](test) run search score TopN case in nonConcurrent group #67363
Open
16 tasks

Navigation Menu

Uh oh!

[fix](fe) Preserve path partition columns with CSV schema - #67407

[fix](fe) Preserve path partition columns with CSV schema#67407
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25094-csv-schema-path-partitions

0AyanamiRei commented Sep 1, 2026•
edited
Loading

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

0AyanamiRei commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

0AyanamiRei commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

0AyanamiRei commented Sep 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

0AyanamiRei commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

0AyanamiRei commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

0AyanamiRei commented Sep 1, 2026•
edited
Loading