[support](orc)support orc file meta cache. by hubgeter · Pull Request #54591 · apache/doris

hubgeter · 2025-08-11T17:23:23Z

What problem does this PR solve?

Problem Summary:
This PR includes three changes:

Support for file meta cache for ORC files.
Changed the file meta cache key from file name + modification time to file name + modification time / file size , reduce the chance of reading old meta.
Removed some unused code in the parquet meta.
Users can use profile to observe whether the cache hits or not.
FileFooterHitCache: hit cache.
FileFooterReadCalls: not hit cache or disable cache.

BTW : disable cache : be conf max_external_file_meta_cache_num <= 0

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-08-11T17:23:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hubgeter · 2025-08-11T17:23:33Z

run buildall

doris-robot · 2025-08-11T18:24:16Z

TPC-H: Total hot run time: 33897 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ec68b36b49ba481d09f17efbeef530c63118888, data reload: false
------ Round 1 ----------------------------------
q1	17609	5243	5093	5093
q2	1916	289	177	177
q3	10310	1266	748	748
q4	10209	990	534	534
q5	7489	2456	2313	2313
q6	172	157	130	130
q7	918	739	628	628
q8	9319	1307	1090	1090
q9	7085	5359	5152	5152
q10	6963	2389	1943	1943
q11	473	291	265	265
q12	342	363	225	225
q13	17770	3662	3000	3000
q14	243	230	208	208
q15	571	469	490	469
q16	433	421	372	372
q17	613	856	359	359
q18	7409	7078	7048	7048
q19	1091	956	569	569
q20	347	323	219	219
q21	4125	3242	2375	2375
q22	1095	1024	980	980
Total cold run time: 106502 ms
Total hot run time: 33897 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5173	5453	5116	5116
q2	243	321	217	217
q3	2187	2671	2290	2290
q4	1372	1745	1327	1327
q5	4213	4574	4498	4498
q6	220	166	135	135
q7	2069	1928	1823	1823
q8	2641	2648	2591	2591
q9	7335	7317	7244	7244
q10	3080	3233	2997	2997
q11	562	510	497	497
q12	708	796	632	632
q13	3496	3887	3281	3281
q14	314	318	288	288
q15	520	469	470	469
q16	432	505	470	470
q17	1195	1562	1405	1405
q18	7879	7939	7520	7520
q19	836	847	981	847
q20	2076	2004	1891	1891
q21	4865	4481	4414	4414
q22	1117	1017	1001	1001
Total cold run time: 52533 ms
Total hot run time: 50953 ms

doris-robot · 2025-08-11T18:35:46Z

TPC-DS: Total hot run time: 184447 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ec68b36b49ba481d09f17efbeef530c63118888, data reload: false
query1	984	384	408	384
query2	6530	1722	1756	1722
query3	6739	221	220	220
query4	26667	23336	22943	22943
query5	4307	626	482	482
query6	327	227	199	199
query7	4638	516	297	297
query8	276	238	214	214
query9	8602	2864	2856	2856
query10	461	324	277	277
query11	15672	14907	15140	14907
query12	160	114	113	113
query13	1639	539	404	404
query14	8895	5780	5675	5675
query15	197	180	183	180
query16	7194	613	449	449
query17	1173	697	580	580
query18	1980	395	310	310
query19	200	177	157	157
query20	124	116	118	116
query21	209	123	101	101
query22	4093	4244	3988	3988
query23	34182	33334	33344	33334
query24	8077	2384	2355	2355
query25	535	468	395	395
query26	912	281	159	159
query27	2736	492	342	342
query28	4380	2250	2188	2188
query29	696	553	456	456
query30	299	220	190	190
query31	890	816	710	710
query32	82	77	71	71
query33	533	389	356	356
query34	797	831	520	520
query35	798	849	748	748
query36	991	1038	889	889
query37	121	107	84	84
query38	4026	3985	3908	3908
query39	1496	1416	1377	1377
query40	221	131	118	118
query41	67	63	61	61
query42	119	116	115	115
query43	487	534	465	465
query44	1335	856	856	856
query45	178	176	173	173
query46	868	1021	653	653
query47	1770	1786	1717	1717
query48	397	435	334	334
query49	694	493	431	431
query50	657	690	404	404
query51	4060	4185	4076	4076
query52	116	110	103	103
query53	236	263	195	195
query54	582	598	513	513
query55	85	84	84	84
query56	324	299	296	296
query57	1183	1187	1122	1122
query58	275	263	263	263
query59	2682	2780	2548	2548
query60	347	340	329	329
query61	129	125	118	118
query62	799	759	648	648
query63	228	196	195	195
query64	3508	1085	677	677
query65	4285	4173	4157	4157
query66	1023	431	314	314
query67	15460	15192	15039	15039
query68	7978	913	568	568
query69	481	318	283	283
query70	1232	1170	1125	1125
query71	458	325	314	314
query72	5591	4728	4913	4728
query73	725	689	355	355
query74	9237	8778	8963	8778
query75	3823	3085	2637	2637
query76	3620	1131	749	749
query77	867	396	319	319
query78	9458	9732	8872	8872
query79	2179	807	599	599
query80	614	557	483	483
query81	470	265	220	220
query82	423	129	111	111
query83	270	255	233	233
query84	245	108	89	89
query85	787	376	346	346
query86	398	303	291	291
query87	4264	4303	4106	4106
query88	3302	2226	2201	2201
query89	379	339	288	288
query90	1929	223	225	223
query91	134	141	114	114
query92	93	68	67	67
query93	1400	997	643	643
query94	672	407	313	313
query95	397	324	308	308
query96	484	574	278	278
query97	2639	2694	2599	2599
query98	260	218	236	218
query99	1652	1407	1329	1329
Total cold run time: 271194 ms
Total hot run time: 184447 ms

doris-robot · 2025-08-11T18:41:14Z

ClickBench: Total hot run time: 32.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ec68b36b49ba481d09f17efbeef530c63118888, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.08	0.07
query4	1.64	0.11	0.11
query5	0.43	0.41	0.40
query6	1.16	0.64	0.65
query7	0.03	0.02	0.02
query8	0.04	0.04	0.04
query9	0.60	0.51	0.51
query10	0.57	0.57	0.58
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.64	0.60	0.61
query14	0.80	0.82	0.84
query15	0.88	0.86	0.85
query16	0.38	0.41	0.39
query17	1.04	1.06	1.06
query18	0.21	0.20	0.20
query19	1.92	1.81	1.89
query20	0.01	0.01	0.02
query21	15.39	0.93	0.55
query22	0.77	1.14	0.71
query23	14.94	1.39	0.63
query24	6.97	1.00	0.69
query25	0.55	0.22	0.10
query26	0.54	0.15	0.13
query27	0.06	0.05	0.05
query28	9.82	0.92	0.44
query29	12.60	3.95	3.29
query30	3.15	3.01	2.94
query31	2.83	0.59	0.38
query32	3.23	0.57	0.47
query33	3.09	3.14	3.10
query34	15.84	5.47	4.84
query35	4.90	4.88	5.02
query36	0.69	0.52	0.49
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.19	0.15	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.03	0.04	0.03
Total cold run time: 106.9 s
Total hot run time: 32.46 s

hubgeter · 2025-08-12T03:58:19Z

- return buf.str();
-}
-
-void FieldDescriptor::iceberg_sanitize(const std::vector<std::string>& read_columns) {


This code is currently unused.
reason :
Prior to pull request #27108, the Iceberg Parquet reader parsed the field ID and file name from the Parquet file's Properties iceberg.schema.
Since iceberg.schema stores the table name, which is not in valid Avro format, the PR #27108 needs to convert it (sanitize_avro_name).
In the current master code implementation, the field ID comes from the schema.

hubgeter · 2025-08-12T17:39:53Z

run buildall

doris-robot · 2025-08-12T18:13:51Z

TPC-H: Total hot run time: 33906 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9da60e4583a8ae6846eb9ce78b6b51cada2f5749, data reload: false
------ Round 1 ----------------------------------
q1	17568	5275	5081	5081
q2	1908	281	182	182
q3	10320	1358	716	716
q4	10208	1036	519	519
q5	7552	2384	2366	2366
q6	176	158	128	128
q7	905	751	608	608
q8	9302	1281	1055	1055
q9	6999	5333	5147	5147
q10	6892	2357	1947	1947
q11	478	288	269	269
q12	350	352	223	223
q13	17761	3593	2983	2983
q14	234	235	217	217
q15	541	486	474	474
q16	435	421	379	379
q17	608	836	371	371
q18	7398	7199	7117	7117
q19	1085	946	557	557
q20	357	329	218	218
q21	4056	2589	2376	2376
q22	1110	1050	973	973
Total cold run time: 106243 ms
Total hot run time: 33906 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5132	5120	5108	5108
q2	241	319	215	215
q3	2167	2660	2273	2273
q4	1389	1791	1325	1325
q5	4218	4444	4527	4444
q6	218	173	127	127
q7	2088	1935	1834	1834
q8	2563	2708	2549	2549
q9	7285	7168	7495	7168
q10	3107	3252	2892	2892
q11	576	518	480	480
q12	724	779	655	655
q13	3471	3897	3337	3337
q14	293	289	304	289
q15	530	477	494	477
q16	462	483	479	479
q17	1205	1552	1373	1373
q18	7931	7673	7576	7576
q19	778	769	882	769
q20	1913	1940	1821	1821
q21	4757	4414	4439	4414
q22	1072	1019	992	992
Total cold run time: 52120 ms
Total hot run time: 50597 ms

doris-robot · 2025-08-12T18:25:23Z

TPC-DS: Total hot run time: 184961 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9da60e4583a8ae6846eb9ce78b6b51cada2f5749, data reload: false
query1	1009	405	399	399
query2	6519	1703	1702	1702
query3	6739	220	217	217
query4	26350	24006	23086	23086
query5	4292	587	490	490
query6	323	224	197	197
query7	4618	487	282	282
query8	274	228	209	209
query9	8796	2846	2843	2843
query10	470	322	281	281
query11	15657	14913	14765	14765
query12	163	117	108	108
query13	1674	540	423	423
query14	8593	5665	5689	5665
query15	210	190	169	169
query16	7300	655	496	496
query17	1220	725	619	619
query18	1992	429	326	326
query19	205	196	168	168
query20	123	127	117	117
query21	217	130	102	102
query22	4287	4399	4073	4073
query23	34363	33295	33312	33295
query24	8197	2411	2335	2335
query25	553	468	393	393
query26	1250	267	156	156
query27	2748	503	329	329
query28	4322	2235	2192	2192
query29	775	608	438	438
query30	280	221	187	187
query31	885	794	702	702
query32	78	73	67	67
query33	544	366	335	335
query34	789	840	486	486
query35	787	813	768	768
query36	981	1000	926	926
query37	122	109	84	84
query38	4037	4009	3995	3995
query39	1463	1579	1405	1405
query40	220	124	112	112
query41	57	56	64	56
query42	119	109	107	107
query43	490	480	470	470
query44	1305	838	842	838
query45	173	172	179	172
query46	854	1004	641	641
query47	1792	1848	1770	1770
query48	375	408	308	308
query49	709	476	374	374
query50	638	696	397	397
query51	4131	4089	4022	4022
query52	118	115	104	104
query53	232	255	193	193
query54	576	578	524	524
query55	87	84	84	84
query56	302	317	305	305
query57	1184	1196	1130	1130
query58	279	257	266	257
query59	2653	2695	2589	2589
query60	341	329	311	311
query61	130	123	119	119
query62	815	728	674	674
query63	227	189	183	183
query64	4376	1081	677	677
query65	4293	4216	4224	4216
query66	1160	420	315	315
query67	15607	15246	15258	15246
query68	9019	906	559	559
query69	465	322	278	278
query70	1206	1157	1155	1155
query71	449	321	307	307
query72	5617	4828	4989	4828
query73	788	723	351	351
query74	9359	9134	8926	8926
query75	4244	3054	2619	2619
query76	3661	1122	720	720
query77	816	394	318	318
query78	9447	9878	8838	8838
query79	1579	822	598	598
query80	599	542	473	473
query81	465	260	221	221
query82	188	136	108	108
query83	279	250	244	244
query84	294	107	83	83
query85	764	366	331	331
query86	332	308	295	295
query87	4259	4262	4200	4200
query88	2775	2191	2193	2191
query89	379	323	282	282
query90	2072	225	217	217
query91	145	144	109	109
query92	85	70	65	65
query93	1134	999	640	640
query94	665	394	305	305
query95	387	318	304	304
query96	477	605	275	275
query97	2631	2682	2545	2545
query98	221	225	215	215
query99	1418	1397	1337	1337
Total cold run time: 272667 ms
Total hot run time: 184961 ms

doris-robot · 2025-08-12T18:30:52Z

ClickBench: Total hot run time: 32.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9da60e4583a8ae6846eb9ce78b6b51cada2f5749, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.24	0.07	0.08
query4	1.62	0.11	0.11
query5	0.44	0.42	0.42
query6	1.18	0.65	0.65
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.60	0.52	0.52
query10	0.58	0.58	0.57
query11	0.16	0.11	0.11
query12	0.16	0.11	0.12
query13	0.62	0.61	0.61
query14	0.79	0.84	0.84
query15	0.88	0.86	0.88
query16	0.38	0.40	0.38
query17	1.02	1.03	1.05
query18	0.21	0.19	0.20
query19	1.93	1.81	1.84
query20	0.02	0.01	0.01
query21	15.39	0.94	0.54
query22	0.77	1.12	0.74
query23	14.95	1.39	0.64
query24	6.51	1.44	1.09
query25	0.47	0.22	0.10
query26	0.62	0.15	0.13
query27	0.05	0.06	0.05
query28	9.55	0.92	0.42
query29	12.57	3.94	3.23
query30	3.08	3.06	3.03
query31	2.81	0.59	0.39
query32	3.23	0.58	0.47
query33	3.03	3.16	3.07
query34	15.93	5.55	4.89
query35	4.91	4.84	4.97
query36	0.67	0.51	0.49
query37	0.10	0.07	0.07
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.08 s
Total hot run time: 32.91 s

hubgeter · 2025-08-13T06:55:35Z

run buildall

doris-robot · 2025-08-13T07:37:24Z

TPC-H: Total hot run time: 33751 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a13b583b0d727d6f7cd0a93c5f9e700404dae6df, data reload: false
------ Round 1 ----------------------------------
q1	17586	5176	5007	5007
q2	1918	296	190	190
q3	10294	1297	703	703
q4	10206	1004	527	527
q5	7774	2437	2329	2329
q6	190	164	130	130
q7	892	752	608	608
q8	9304	1293	1112	1112
q9	7147	5236	5126	5126
q10	6955	2366	1954	1954
q11	483	281	275	275
q12	356	345	223	223
q13	17769	3601	3005	3005
q14	226	239	218	218
q15	554	478	493	478
q16	418	429	364	364
q17	608	874	364	364
q18	7956	7149	7019	7019
q19	1565	966	549	549
q20	339	332	226	226
q21	3831	3174	2370	2370
q22	1065	1047	974	974
Total cold run time: 107436 ms
Total hot run time: 33751 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5249	5065	5095	5065
q2	237	329	221	221
q3	2187	2675	2279	2279
q4	1354	1776	1351	1351
q5	4241	4614	4545	4545
q6	224	172	154	154
q7	2031	1987	1841	1841
q8	2597	2603	2538	2538
q9	7283	7414	7183	7183
q10	3090	3261	2873	2873
q11	588	533	500	500
q12	689	852	676	676
q13	3500	3903	3502	3502
q14	295	302	270	270
q15	520	467	482	467
q16	451	488	461	461
q17	1182	1542	1328	1328
q18	7956	7692	7557	7557
q19	835	827	850	827
q20	2035	2128	1890	1890
q21	4986	4470	4280	4280
q22	1066	1036	1005	1005
Total cold run time: 52596 ms
Total hot run time: 50813 ms

doris-robot · 2025-08-13T07:48:55Z

TPC-DS: Total hot run time: 184313 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a13b583b0d727d6f7cd0a93c5f9e700404dae6df, data reload: false
query1	990	396	400	396
query2	6502	1732	1715	1715
query3	6742	225	223	223
query4	26200	23371	22953	22953
query5	4341	628	489	489
query6	313	221	204	204
query7	4630	498	286	286
query8	290	227	209	209
query9	8634	2898	2871	2871
query10	457	312	298	298
query11	15771	15035	14733	14733
query12	163	119	114	114
query13	1659	544	428	428
query14	9031	5727	5632	5632
query15	204	191	169	169
query16	7267	624	484	484
query17	1227	719	635	635
query18	2004	419	320	320
query19	193	194	162	162
query20	135	122	120	120
query21	216	121	108	108
query22	4381	4283	4184	4184
query23	34294	33154	33300	33154
query24	8076	2308	2317	2308
query25	547	467	395	395
query26	1233	267	157	157
query27	2733	505	354	354
query28	4335	2254	2232	2232
query29	768	550	444	444
query30	282	212	204	204
query31	904	780	703	703
query32	81	74	75	74
query33	574	369	337	337
query34	792	839	511	511
query35	788	809	742	742
query36	957	1002	878	878
query37	122	104	82	82
query38	4105	3971	3950	3950
query39	1454	1445	1396	1396
query40	211	124	113	113
query41	62	56	53	53
query42	118	107	108	107
query43	491	483	462	462
query44	1326	840	850	840
query45	175	172	163	163
query46	865	1006	641	641
query47	1800	1843	1759	1759
query48	385	432	308	308
query49	718	483	382	382
query50	646	682	388	388
query51	4047	4169	4135	4135
query52	114	108	100	100
query53	231	268	194	194
query54	586	579	528	528
query55	87	89	83	83
query56	306	310	291	291
query57	1224	1220	1139	1139
query58	280	263	273	263
query59	2597	2728	2638	2638
query60	342	330	322	322
query61	127	122	116	116
query62	799	744	657	657
query63	227	190	187	187
query64	4361	1004	749	749
query65	4283	4202	4213	4202
query66	1155	406	323	323
query67	15529	15309	15067	15067
query68	8106	898	581	581
query69	465	336	287	287
query70	1221	1112	1085	1085
query71	440	334	305	305
query72	5524	4750	4759	4750
query73	725	603	351	351
query74	8889	9117	8602	8602
query75	3779	3053	2623	2623
query76	3667	1140	727	727
query77	782	401	326	326
query78	9580	9818	8868	8868
query79	2182	830	579	579
query80	592	543	481	481
query81	469	256	221	221
query82	425	172	106	106
query83	251	247	234	234
query84	244	97	83	83
query85	788	373	336	336
query86	334	328	275	275
query87	4188	4329	4172	4172
query88	3125	2196	2175	2175
query89	392	312	286	286
query90	1921	219	218	218
query91	139	138	158	138
query92	82	72	69	69
query93	1161	988	633	633
query94	664	399	303	303
query95	398	314	309	309
query96	486	569	269	269
query97	2652	2696	2578	2578
query98	237	210	214	210
query99	1557	1401	1269	1269
Total cold run time: 271952 ms
Total hot run time: 184313 ms

doris-robot · 2025-08-13T07:54:23Z

ClickBench: Total hot run time: 32.26 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a13b583b0d727d6f7cd0a93c5f9e700404dae6df, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.08	0.08
query4	1.61	0.11	0.11
query5	0.42	0.41	0.41
query6	1.16	0.63	0.65
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.61	0.51	0.51
query10	0.56	0.57	0.57
query11	0.15	0.10	0.10
query12	0.15	0.11	0.12
query13	0.63	0.62	0.61
query14	0.80	0.81	0.83
query15	0.88	0.84	0.84
query16	0.38	0.40	0.38
query17	1.03	1.04	1.04
query18	0.21	0.20	0.20
query19	1.95	1.85	1.84
query20	0.02	0.01	0.01
query21	15.48	0.95	0.55
query22	0.78	1.17	0.73
query23	14.93	1.41	0.62
query24	7.08	0.67	1.45
query25	0.49	0.10	0.05
query26	0.61	0.17	0.14
query27	0.05	0.06	0.05
query28	9.73	0.91	0.43
query29	12.53	3.90	3.23
query30	3.09	3.05	2.92
query31	2.83	0.57	0.38
query32	3.25	0.56	0.46
query33	3.09	3.07	3.17
query34	15.91	5.43	4.86
query35	4.90	4.91	4.93
query36	0.70	0.50	0.49
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.15	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.95 s
Total hot run time: 32.26 s

doris-robot · 2025-08-13T07:55:51Z

BE UT Coverage Report

Increment line coverage 57.32% (94/164) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.09% (16646/28171)
Line Coverage	47.94% (150961/314926)
Region Coverage	36.74% (113007/307580)
Branch Coverage	39.67% (50164/126451)

hello-stephen · 2025-08-13T09:05:52Z

BE Regression && UT Coverage Report

Increment line coverage 66.46% (109/164) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.75% (22603/27648)
Line Coverage	74.34% (234125/314933)
Region Coverage	61.63% (194661/315841)
Branch Coverage	65.69% (84080/127997)

morningman · 2025-08-15T17:43:20Z

 : _file_reader;
 }
+ if (_file_metadata) {
+ std::cout << "_file_metadata not null\n";


Remove this

hubgeter · 2025-08-19T17:12:37Z

run buildall

doris-robot · 2025-08-19T17:39:43Z

TPC-H: Total hot run time: 33785 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6aea5bc56ad1aaf943dec5878d87dd17294e517, data reload: false
------ Round 1 ----------------------------------
q1	17611	5253	5120	5120
q2	1922	291	204	204
q3	10287	1298	720	720
q4	10217	981	532	532
q5	7514	2423	2257	2257
q6	176	158	127	127
q7	887	729	620	620
q8	9307	1246	1030	1030
q9	6980	5068	5118	5068
q10	6931	2393	1964	1964
q11	481	286	272	272
q12	349	351	211	211
q13	17782	3681	3058	3058
q14	229	237	204	204
q15	557	485	479	479
q16	420	417	364	364
q17	598	860	347	347
q18	7680	7144	7098	7098
q19	1198	944	553	553
q20	344	324	211	211
q21	3779	3186	2358	2358
q22	1062	1043	988	988
Total cold run time: 106311 ms
Total hot run time: 33785 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5212	5105	5096	5096
q2	239	325	223	223
q3	2161	2678	2292	2292
q4	1359	1778	1350	1350
q5	4229	4500	4555	4500
q6	205	160	127	127
q7	2097	1974	1816	1816
q8	2925	2479	2541	2479
q9	7285	7307	7177	7177
q10	3147	3322	2883	2883
q11	584	522	505	505
q12	707	784	589	589
q13	3555	3896	3332	3332
q14	310	320	288	288
q15	537	488	471	471
q16	431	504	440	440
q17	1179	1477	1438	1438
q18	7782	7632	7702	7632
q19	793	805	877	805
q20	1882	1963	1809	1809
q21	4674	4389	4333	4333
q22	1058	1032	1023	1023
Total cold run time: 52351 ms
Total hot run time: 50608 ms

doris-robot · 2025-08-19T17:51:16Z

TPC-DS: Total hot run time: 184179 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6aea5bc56ad1aaf943dec5878d87dd17294e517, data reload: false
query1	974	383	421	383
query2	6518	1711	1721	1711
query3	6737	221	222	221
query4	26302	23468	23043	23043
query5	4333	622	474	474
query6	294	210	188	188
query7	4623	510	298	298
query8	287	249	224	224
query9	8897	3067	2952	2952
query10	487	347	285	285
query11	15914	14970	14728	14728
query12	161	125	116	116
query13	1657	534	423	423
query14	9381	5788	5754	5754
query15	203	184	173	173
query16	7593	659	488	488
query17	1200	761	596	596
query18	2028	408	332	332
query19	214	200	176	176
query20	126	118	121	118
query21	210	125	110	110
query22	4236	4249	3883	3883
query23	34210	33289	33219	33219
query24	8130	2352	2348	2348
query25	533	466	407	407
query26	1238	268	159	159
query27	2713	495	333	333
query28	4339	2188	2179	2179
query29	733	605	435	435
query30	285	219	185	185
query31	898	797	710	710
query32	88	74	74	74
query33	562	377	350	350
query34	796	828	502	502
query35	840	807	746	746
query36	967	1025	926	926
query37	125	107	86	86
query38	3965	4002	4014	4002
query39	1477	1447	1402	1402
query40	219	126	119	119
query41	59	55	54	54
query42	124	109	110	109
query43	493	477	478	477
query44	1314	840	844	840
query45	177	173	163	163
query46	845	995	630	630
query47	1744	1797	1707	1707
query48	380	413	309	309
query49	731	494	389	389
query50	645	683	394	394
query51	4054	4121	4006	4006
query52	113	118	107	107
query53	233	262	193	193
query54	597	594	519	519
query55	88	94	83	83
query56	314	299	292	292
query57	1180	1173	1123	1123
query58	291	292	264	264
query59	2579	2630	2557	2557
query60	341	344	343	343
query61	128	124	126	124
query62	837	727	666	666
query63	226	190	189	189
query64	4283	1056	698	698
query65	4286	4200	4217	4200
query66	1119	421	335	335
query67	15408	15515	14922	14922
query68	8677	927	575	575
query69	466	333	284	284
query70	1262	1114	1102	1102
query71	468	323	305	305
query72	5313	4667	4642	4642
query73	716	583	349	349
query74	9166	9116	8795	8795
query75	4034	3064	2640	2640
query76	3638	1149	718	718
query77	875	398	323	323
query78	9621	9626	8852	8852
query79	2344	815	593	593
query80	603	545	464	464
query81	476	260	222	222
query82	450	141	110	110
query83	287	271	242	242
query84	293	109	87	87
query85	794	388	335	335
query86	356	314	293	293
query87	4305	4274	4187	4187
query88	3254	2207	2188	2188
query89	392	325	283	283
query90	1934	220	221	220
query91	139	140	156	140
query92	84	75	66	66
query93	1713	1014	649	649
query94	674	401	313	313
query95	394	309	307	307
query96	489	587	275	275
query97	2595	2669	2557	2557
query98	245	220	213	213
query99	1425	1385	1299	1299
Total cold run time: 274564 ms
Total hot run time: 184179 ms

doris-robot · 2025-08-19T17:56:44Z

ClickBench: Total hot run time: 32.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6aea5bc56ad1aaf943dec5878d87dd17294e517, data reload: false
query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.08	0.08
query4	1.62	0.11	0.11
query5	0.44	0.42	0.39
query6	1.18	0.65	0.65
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.60	0.53	0.50
query10	0.59	0.57	0.57
query11	0.16	0.11	0.11
query12	0.16	0.12	0.11
query13	0.62	0.61	0.62
query14	0.81	0.83	0.83
query15	0.90	0.86	0.87
query16	0.40	0.39	0.38
query17	1.02	1.04	1.01
query18	0.21	0.20	0.20
query19	1.92	1.87	1.89
query20	0.02	0.01	0.01
query21	15.41	0.99	0.57
query22	0.80	1.07	0.74
query23	14.99	1.37	0.63
query24	6.88	1.19	0.88
query25	0.50	0.19	0.08
query26	0.61	0.16	0.14
query27	0.05	0.05	0.05
query28	9.96	0.90	0.42
query29	12.55	3.90	3.25
query30	3.02	3.02	2.96
query31	2.84	0.59	0.38
query32	3.23	0.54	0.47
query33	2.97	3.19	3.11
query34	16.21	5.48	4.89
query35	4.91	4.87	5.00
query36	0.69	0.51	0.50
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.13	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.25 s
Total hot run time: 32.7 s

hubgeter · 2025-08-20T07:07:01Z

run buildall

doris-robot · 2025-08-20T07:26:43Z

TPC-H: Total hot run time: 34002 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c97cbc4b8841581f3359da01cfd283f44a4fc8a, data reload: false
------ Round 1 ----------------------------------
q1	17585	5296	5133	5133
q2	1945	282	175	175
q3	10307	1329	710	710
q4	10219	1030	534	534
q5	7543	2363	2399	2363
q6	185	158	132	132
q7	881	742	607	607
q8	9291	1297	1153	1153
q9	6895	5158	5126	5126
q10	6872	2375	2006	2006
q11	474	288	271	271
q12	338	351	208	208
q13	17787	3633	2999	2999
q14	235	232	217	217
q15	549	485	484	484
q16	422	418	366	366
q17	592	855	357	357
q18	7373	7073	7220	7073
q19	1082	940	560	560
q20	348	331	228	228
q21	3924	2531	2326	2326
q22	1071	1018	974	974
Total cold run time: 105918 ms
Total hot run time: 34002 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5192	5142	5101	5101
q2	240	322	217	217
q3	2141	2665	2279	2279
q4	1370	1800	1340	1340
q5	4197	4446	4543	4446
q6	225	179	130	130
q7	2060	1938	1843	1843
q8	2606	2678	2669	2669
q9	7388	7272	7312	7272
q10	3073	3276	2861	2861
q11	577	507	494	494
q12	698	799	639	639
q13	3624	3944	3325	3325
q14	288	328	290	290
q15	532	482	482	482
q16	451	497	460	460
q17	1190	1613	1385	1385
q18	8062	7791	7607	7607
q19	809	893	893	893
q20	2026	1999	1865	1865
q21	4797	4384	4181	4181
q22	1081	1025	1003	1003
Total cold run time: 52627 ms
Total hot run time: 50782 ms

doris-robot · 2025-08-20T07:38:09Z

TPC-DS: Total hot run time: 185159 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c97cbc4b8841581f3359da01cfd283f44a4fc8a, data reload: false
query1	978	389	403	389
query2	6541	1744	1788	1744
query3	6747	223	220	220
query4	26190	23249	23480	23249
query5	4344	649	500	500
query6	316	216	201	201
query7	4629	512	296	296
query8	280	230	221	221
query9	8575	2882	2860	2860
query10	478	342	295	295
query11	15493	15015	14754	14754
query12	164	114	121	114
query13	1655	547	406	406
query14	9127	5783	5809	5783
query15	203	181	159	159
query16	7663	652	501	501
query17	1170	706	572	572
query18	2009	399	308	308
query19	191	183	163	163
query20	136	120	124	120
query21	206	120	103	103
query22	4098	4174	3969	3969
query23	34535	33263	33364	33263
query24	8166	2389	2349	2349
query25	524	451	423	423
query26	1228	268	162	162
query27	2747	508	351	351
query28	4332	2228	2195	2195
query29	737	587	450	450
query30	285	224	191	191
query31	897	807	689	689
query32	82	73	71	71
query33	546	374	337	337
query34	798	844	499	499
query35	810	825	743	743
query36	988	1006	933	933
query37	122	109	92	92
query38	4142	3938	3987	3938
query39	1501	1408	1401	1401
query40	221	129	121	121
query41	62	55	52	52
query42	124	109	119	109
query43	508	525	481	481
query44	1364	873	873	873
query45	182	165	166	165
query46	866	1016	656	656
query47	1758	1819	1716	1716
query48	391	409	321	321
query49	703	499	380	380
query50	644	694	402	402
query51	4118	4171	4044	4044
query52	118	113	101	101
query53	243	268	199	199
query54	597	586	516	516
query55	90	88	84	84
query56	320	309	303	303
query57	1179	1205	1122	1122
query58	288	272	263	263
query59	2643	2661	2566	2566
query60	344	339	339	339
query61	126	125	118	118
query62	783	764	651	651
query63	236	195	189	189
query64	4248	1006	700	700
query65	4342	4243	4231	4231
query66	1090	440	416	416
query67	15587	15200	14896	14896
query68	5056	956	584	584
query69	477	323	295	295
query70	1165	1162	1134	1134
query71	400	326	313	313
query72	5522	4870	5026	4870
query73	689	674	354	354
query74	9207	8825	9001	8825
query75	3157	3087	2609	2609
query76	3245	1145	741	741
query77	495	412	317	317
query78	9543	9767	8868	8868
query79	1022	883	610	610
query80	632	578	499	499
query81	490	258	229	229
query82	372	144	105	105
query83	252	260	240	240
query84	249	109	95	95
query85	760	413	335	335
query86	339	325	322	322
query87	4254	4344	4228	4228
query88	2868	2245	2235	2235
query89	390	320	293	293
query90	1755	250	233	233
query91	137	140	108	108
query92	75	68	68	68
query93	1141	1014	651	651
query94	624	415	303	303
query95	400	310	311	310
query96	482	598	281	281
query97	2681	2656	2597	2597
query98	245	217	211	211
query99	1345	1400	1268	1268
Total cold run time: 266277 ms
Total hot run time: 185159 ms

doris-robot · 2025-08-20T07:43:37Z

ClickBench: Total hot run time: 32.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c97cbc4b8841581f3359da01cfd283f44a4fc8a, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.06	0.07
query4	1.62	0.11	0.11
query5	0.42	0.42	0.40
query6	1.19	0.64	0.67
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.53	0.52
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.60
query14	0.80	0.83	0.85
query15	0.88	0.85	0.86
query16	0.41	0.41	0.38
query17	1.07	1.03	1.04
query18	0.21	0.19	0.20
query19	1.92	1.82	1.76
query20	0.02	0.01	0.01
query21	15.40	0.95	0.58
query22	0.78	1.17	0.77
query23	14.77	1.35	0.64
query24	6.91	0.73	1.37
query25	0.53	0.28	0.07
query26	0.58	0.15	0.14
query27	0.06	0.05	0.05
query28	9.86	0.95	0.44
query29	12.57	3.87	3.25
query30	3.07	3.06	2.93
query31	2.82	0.58	0.37
query32	3.25	0.55	0.48
query33	3.06	3.13	3.03
query34	16.10	5.49	4.89
query35	4.96	4.94	4.95
query36	0.71	0.50	0.48
query37	0.09	0.07	0.06
query38	0.06	0.05	0.04
query39	0.03	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.99 s
Total hot run time: 32.41 s

doris-robot · 2025-08-20T08:07:46Z

BE UT Coverage Report

Increment line coverage 61.90% (117/189) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.64% (16797/28164)
Line Coverage	48.55% (152766/314659)
Region Coverage	37.50% (116098/309635)
Branch Coverage	40.41% (51256/126838)

dataroaring

LGTM

github-actions · 2025-08-22T07:47:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-08-22T07:47:19Z

PR approved by anyone and no changes requested.

morningman · 2025-08-28T01:57:15Z

run check_coverage

Problem Summary: This PR includes three changes: 1. Support for file meta cache for ORC files. 2. Changed the file meta cache key from `file name + modification time` to `file name + modification time / file size` , reduce the chance of reading old meta. 3. Removed some unused code in the parquet meta. 4. Users can use profile to observe whether the cache hits or not. `FileFooterHitCache`: hit cache. `FileFooterReadCalls`: not hit cache or disable cache. BTW : disable cache : be conf `max_external_file_meta_cache_num` <= 0

bp #54591

hubgeter commented Aug 12, 2025
View reviewed changes

hubgeter marked this pull request as ready for review August 12, 2025 17:39

hubgeter requested review from dataroaring and gavinchou as code owners August 12, 2025 17:39

morningman added the dev/3.1.x label Aug 15, 2025

morningman reviewed Aug 15, 2025
View reviewed changes

[support](orc)support orc file meta cache.
b6aea5b

hubgeter force-pushed the support_orc_cache branch from cbc1456 to b6aea5bCompare August 19, 2025 17:12

fix ut
6c97cbc

dataroaring approved these changes Aug 22, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2025

github-actionsBot added the reviewed label Aug 22, 2025

morningman approved these changes Aug 24, 2025
View reviewed changes

gavinchou approved these changes Sep 2, 2025
View reviewed changes

morningman merged commit 9aceb34 into apache:masterSep 2, 2025
28 of 30 checks passed

github-actionsBot added the dev/3.1.x-conflict label Sep 2, 2025

hubgeter mentioned this pull request Sep 2, 2025
branch-3.1: [support](orc)support orc file meta cache. #54591 #55584
Merged

morrySnow pushed a commit that referenced this pull request Sep 4, 2025
branch-3.1: [support](orc)support orc file meta cache.#54591(#55584)
c1d4af2
bp #54591

morningman added dev/3.1.1-merged and removed dev/3.1.x dev/3.1.x-conflict labels Sep 11, 2025

morrySnow mentioned this pull request Sep 22, 2025
Release Note 3.1.1 #56289
Closed

Uh oh!

Conversation

hubgeter commented Aug 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Aug 11, 2025

Uh oh!

hubgeter commented Aug 11, 2025

Uh oh!

doris-robot commented Aug 11, 2025

Uh oh!

doris-robot commented Aug 11, 2025

Uh oh!

doris-robot commented Aug 11, 2025

Uh oh!

hubgeterAug 12, 2025

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Aug 12, 2025

Uh oh!

doris-robot commented Aug 12, 2025

Uh oh!

doris-robot commented Aug 12, 2025

Uh oh!

doris-robot commented Aug 12, 2025

Uh oh!

hubgeter commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

Uh oh!

doris-robot commented Aug 13, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 13, 2025

BE Regression && UT Coverage Report

Uh oh!

morningmanAug 15, 2025

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Aug 19, 2025

Uh oh!

doris-robot commented Aug 19, 2025

Uh oh!

doris-robot commented Aug 19, 2025

Uh oh!

doris-robot commented Aug 19, 2025

Uh oh!

hubgeter commented Aug 20, 2025

Uh oh!

doris-robot commented Aug 20, 2025

Uh oh!

doris-robot commented Aug 20, 2025

Uh oh!

doris-robot commented Aug 20, 2025

Uh oh!

doris-robot commented Aug 20, 2025

BE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 22, 2025

Uh oh!

github-actionsBot commented Aug 22, 2025

Uh oh!

morningman commented Aug 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

hubgeter commented Aug 11, 2025•
edited
Loading