[fix](cloud) release fd after clear file cache by freemandealer · Pull Request #50555 · apache/doris

freemandealer · 2025-04-29T14:15:30Z

originally, fd is still hold by FDCache after clear file cache (sync) will cause disk space never reaching zero

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-29T14:15:38Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

originally, fd is still hold by FDCache after clear file cache (sync) will cause disk space never reaching zero Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer · 2025-04-29T14:18:33Z

run buildall

doris-robot · 2025-04-29T15:49:34Z

TPC-H: Total hot run time: 34183 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4ccae6df9cab00cd9e10e3192b1942ffaddd580, data reload: false
------ Round 1 ----------------------------------
q1	26139	5055	5056	5055
q2	2081	277	178	178
q3	10533	1231	702	702
q4	10237	1022	549	549
q5	7907	2423	2345	2345
q6	184	163	132	132
q7	915	744	610	610
q8	9321	1375	1131	1131
q9	6809	5094	5110	5094
q10	6826	2311	1900	1900
q11	485	270	270	270
q12	351	360	217	217
q13	17761	3664	3107	3107
q14	235	220	211	211
q15	542	481	480	480
q16	420	434	375	375
q17	600	876	360	360
q18	7671	7222	7223	7222
q19	1625	975	588	588
q20	336	324	215	215
q21	3943	3346	2464	2464
q22	988	978	987	978
Total cold run time: 115909 ms
Total hot run time: 34183 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5163	5083	5149	5083
q2	234	322	230	230
q3	2167	2694	2303	2303
q4	1361	1805	1412	1412
q5	4471	4441	4437	4437
q6	214	173	132	132
q7	2026	1928	1769	1769
q8	2632	2611	2516	2516
q9	7285	7267	6924	6924
q10	3049	3179	2767	2767
q11	575	512	497	497
q12	649	752	633	633
q13	3499	3850	3231	3231
q14	295	330	298	298
q15	536	481	485	481
q16	447	484	423	423
q17	1152	1586	1355	1355
q18	7771	7507	7483	7483
q19	804	872	870	870
q20	1982	2078	1857	1857
q21	5094	4736	4628	4628
q22	1109	1038	1015	1015
Total cold run time: 52515 ms
Total hot run time: 50344 ms

doris-robot · 2025-04-29T16:01:45Z

TPC-DS: Total hot run time: 192162 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4ccae6df9cab00cd9e10e3192b1942ffaddd580, data reload: false
query1	1442	1089	1060	1060
query2	6244	1804	1778	1778
query3	10956	4699	4395	4395
query4	53846	25488	23563	23563
query5	5185	503	446	446
query6	351	201	192	192
query7	4951	507	291	291
query8	315	258	224	224
query9	5919	2567	2581	2567
query10	438	307	257	257
query11	15008	14961	14818	14818
query12	153	114	107	107
query13	1082	535	397	397
query14	10163	6249	6255	6249
query15	206	199	174	174
query16	7091	675	501	501
query17	1087	746	614	614
query18	1562	413	315	315
query19	202	193	173	173
query20	122	121	125	121
query21	201	126	104	104
query22	4440	4433	4378	4378
query23	34214	33300	33353	33300
query24	6644	2450	2399	2399
query25	461	481	407	407
query26	674	271	151	151
query27	2481	509	337	337
query28	3276	2166	2187	2166
query29	565	569	435	435
query30	275	217	185	185
query31	858	858	763	763
query32	81	65	61	61
query33	473	372	347	347
query34	760	877	538	538
query35	775	838	746	746
query36	975	1008	857	857
query37	117	101	85	85
query38	4119	4208	4132	4132
query39	1500	1465	1436	1436
query40	209	134	106	106
query41	59	54	53	53
query42	120	111	112	111
query43	503	507	465	465
query44	1320	831	833	831
query45	181	173	165	165
query46	851	1014	646	646
query47	1829	1875	1827	1827
query48	378	410	316	316
query49	688	501	412	412
query50	654	697	404	404
query51	4339	4223	4137	4137
query52	105	107	95	95
query53	230	273	188	188
query54	593	585	529	529
query55	93	84	80	80
query56	310	319	319	319
query57	1184	1196	1157	1157
query58	283	268	275	268
query59	2576	2574	2485	2485
query60	341	353	301	301
query61	126	130	143	130
query62	704	714	641	641
query63	224	186	185	185
query64	1460	1007	687	687
query65	4348	4252	4226	4226
query66	725	405	303	303
query67	15939	15509	15427	15427
query68	6939	884	510	510
query69	541	308	261	261
query70	1189	1068	1074	1068
query71	493	321	285	285
query72	5825	4847	5014	4847
query73	1426	698	348	348
query74	9224	9105	8780	8780
query75	3802	3220	2702	2702
query76	4230	1178	745	745
query77	642	370	286	286
query78	10012	9999	9259	9259
query79	2502	804	654	654
query80	630	521	428	428
query81	484	245	222	222
query82	434	130	99	99
query83	367	252	238	238
query84	292	108	96	96
query85	783	339	304	304
query86	396	304	276	276
query87	4362	4460	4398	4398
query88	3204	2221	2182	2182
query89	451	320	278	278
query90	1797	202	203	202
query91	149	143	109	109
query92	75	59	59	59
query93	2030	938	572	572
query94	653	418	308	308
query95	363	283	280	280
query96	481	575	274	274
query97	3229	3180	3066	3066
query98	240	215	207	207
query99	1435	1359	1256	1256
Total cold run time: 298264 ms
Total hot run time: 192162 ms

doris-robot · 2025-04-29T16:06:55Z

ClickBench: Total hot run time: 28.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4ccae6df9cab00cd9e10e3192b1942ffaddd580, data reload: false
query1	0.04	0.03	0.04
query2	0.13	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.59	0.59	0.59
query6	1.21	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.56	0.54	0.50
query10	0.57	0.58	0.56
query11	0.15	0.10	0.11
query12	0.15	0.11	0.11
query13	0.60	0.60	0.60
query14	0.78	0.79	0.79
query15	0.86	0.84	0.84
query16	0.38	0.37	0.39
query17	1.05	1.05	1.01
query18	0.21	0.19	0.20
query19	1.86	1.78	1.84
query20	0.02	0.01	0.01
query21	15.41	0.93	0.56
query22	0.75	1.15	0.63
query23	15.05	1.40	0.63
query24	6.88	1.96	0.44
query25	0.39	0.17	0.06
query26	0.64	0.16	0.12
query27	0.06	0.05	0.04
query28	9.31	0.90	0.44
query29	12.61	3.88	3.26
query30	0.26	0.08	0.07
query31	2.81	0.57	0.39
query32	3.23	0.55	0.46
query33	3.05	3.05	3.03
query34	15.77	5.08	4.47
query35	4.47	4.54	4.46
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 102.89 s
Total hot run time: 28.59 s

doris-robot · 2025-04-29T17:38:00Z

BE UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.81% (14773/26953)
Line Coverage	43.92% (129631/295132)
Region Coverage	42.62% (66164/155226)
Branch Coverage	37.23% (33400/89716)

hello-stephen · 2025-04-29T20:42:10Z

BE Regression P0 && UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.25% (20703/26456)
Line Coverage	71.84% (211670/294646)
Region Coverage	69.99% (127116/181630)
Branch Coverage	63.33% (64772/102276)

gavinchou · 2025-04-30T02:26:32Z

 int64_t disposible_queue_size = _disposable_queue.get_elements_num(cache_lock);
 int64_t ttl_queue_size = _ttl_queue.get_elements_num(cache_lock);
+
+ if (config::clear_fd_after_clear_file_cache) {


no need an extra config, just release them directly after files are deleted.

in case of performance and robust regression. we can remove this config if it's stable.

it seems there is no performance problem, line 2069 auto s = _storage->clear(clear_msg); is blocking and time-consuming job before removing FDs.
to avoid too many knobs, keep it simple if the behavior is determined.
we have too many knobs for file cache, too many.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer · 2025-04-30T03:31:20Z

run buildall

doris-robot · 2025-04-30T03:54:43Z

TPC-H: Total hot run time: 33926 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0863ab4a8661b8b012e564f1cfe781f6fe5cfa89, data reload: false
------ Round 1 ----------------------------------
q1	26086	5034	5010	5010
q2	2067	275	190	190
q3	10389	1222	695	695
q4	10222	987	520	520
q5	7554	2360	2356	2356
q6	178	162	133	133
q7	924	755	604	604
q8	9313	1278	1176	1176
q9	6839	5052	5148	5052
q10	6835	2322	1871	1871
q11	480	296	273	273
q12	354	348	215	215
q13	17760	3741	3067	3067
q14	235	228	216	216
q15	532	485	492	485
q16	414	427	365	365
q17	573	868	355	355
q18	7519	7257	7132	7132
q19	1095	938	566	566
q20	342	342	228	228
q21	4257	3339	2449	2449
q22	1023	1031	968	968
Total cold run time: 114991 ms
Total hot run time: 33926 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5088	5056	5044	5044
q2	231	324	230	230
q3	2178	2681	2294	2294
q4	1378	1780	1327	1327
q5	4562	4483	4437	4437
q6	214	167	125	125
q7	1988	1896	1757	1757
q8	2560	2537	2516	2516
q9	7158	7161	7169	7161
q10	3056	3184	2692	2692
q11	566	519	496	496
q12	644	768	619	619
q13	3508	3813	3288	3288
q14	272	307	292	292
q15	530	478	480	478
q16	446	478	452	452
q17	1129	1564	1368	1368
q18	7734	7680	7345	7345
q19	770	751	813	751
q20	1971	2047	1842	1842
q21	5069	4629	4653	4629
q22	1112	1065	980	980
Total cold run time: 52164 ms
Total hot run time: 50123 ms

doris-robot · 2025-04-30T04:06:15Z

TPC-DS: Total hot run time: 184650 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0863ab4a8661b8b012e564f1cfe781f6fe5cfa89, data reload: false
query1	1008	484	504	484
query2	6577	1783	1758	1758
query3	6750	228	218	218
query4	26732	23605	22929	22929
query5	4392	603	451	451
query6	298	207	185	185
query7	4618	494	286	286
query8	285	239	220	220
query9	8674	2578	2565	2565
query10	493	327	283	283
query11	15479	15042	14827	14827
query12	156	117	108	108
query13	1668	548	389	389
query14	9351	6221	6299	6221
query15	213	199	173	173
query16	7508	677	493	493
query17	1196	718	594	594
query18	2099	415	304	304
query19	195	190	157	157
query20	120	117	118	117
query21	217	129	109	109
query22	4034	4009	4005	4005
query23	33943	32844	32744	32744
query24	8427	2444	2389	2389
query25	553	453	380	380
query26	1234	265	153	153
query27	2741	490	336	336
query28	4385	2081	2076	2076
query29	728	549	422	422
query30	279	202	183	183
query31	955	882	740	740
query32	74	66	61	61
query33	601	356	316	316
query34	797	848	512	512
query35	793	811	744	744
query36	953	974	890	890
query37	116	103	78	78
query38	4128	4164	4085	4085
query39	1448	1390	1401	1390
query40	212	121	111	111
query41	59	55	52	52
query42	125	111	107	107
query43	487	498	461	461
query44	1324	799	788	788
query45	181	176	169	169
query46	840	1006	634	634
query47	1769	1802	1704	1704
query48	380	405	296	296
query49	768	495	426	426
query50	656	669	404	404
query51	4129	4079	4028	4028
query52	112	108	97	97
query53	231	249	185	185
query54	591	607	506	506
query55	84	86	86	86
query56	312	301	278	278
query57	1117	1150	1063	1063
query58	275	255	266	255
query59	2621	2638	2482	2482
query60	330	314	309	309
query61	152	131	124	124
query62	782	751	662	662
query63	222	184	187	184
query64	4282	1001	671	671
query65	4332	4217	4252	4217
query66	1106	416	305	305
query67	15699	15270	15446	15270
query68	8140	882	514	514
query69	518	321	264	264
query70	1210	1150	1121	1121
query71	478	320	299	299
query72	5506	4758	4659	4659
query73	676	590	345	345
query74	8856	9143	8816	8816
query75	3893	3199	2699	2699
query76	3747	1196	750	750
query77	790	363	286	286
query78	9905	10099	9246	9246
query79	2365	800	562	562
query80	600	507	492	492
query81	475	249	213	213
query82	425	125	97	97
query83	282	250	230	230
query84	292	102	80	80
query85	782	398	311	311
query86	333	292	274	274
query87	4419	4413	4320	4320
query88	3169	2179	2181	2179
query89	384	309	275	275
query90	1932	205	216	205
query91	136	147	107	107
query92	79	64	55	55
query93	1475	923	574	574
query94	673	410	315	315
query95	371	285	282	282
query96	477	568	270	270
query97	3155	3230	3100	3100
query98	239	222	197	197
query99	1450	1404	1264	1264
Total cold run time: 274628 ms
Total hot run time: 184650 ms

doris-robot · 2025-04-30T04:11:28Z

ClickBench: Total hot run time: 29.28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0863ab4a8661b8b012e564f1cfe781f6fe5cfa89, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.20	0.11
query5	0.58	0.56	0.54
query6	1.19	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.58	0.58	0.58
query11	0.15	0.11	0.10
query12	0.16	0.11	0.12
query13	0.61	0.59	0.59
query14	0.78	0.79	0.82
query15	0.87	0.84	0.87
query16	0.37	0.39	0.39
query17	1.03	1.05	1.01
query18	0.21	0.20	0.20
query19	1.85	1.76	1.73
query20	0.01	0.01	0.01
query21	15.43	0.86	0.52
query22	0.75	1.13	0.74
query23	14.89	1.38	0.63
query24	7.57	1.17	0.95
query25	0.46	0.18	0.06
query26	0.57	0.17	0.13
query27	0.05	0.05	0.05
query28	9.39	0.84	0.43
query29	12.54	4.06	3.40
query30	0.25	0.09	0.07
query31	2.82	0.57	0.38
query32	3.23	0.54	0.45
query33	3.01	3.05	3.09
query34	15.62	5.13	4.51
query35	4.62	4.53	4.51
query36	0.68	0.50	0.48
query37	0.09	0.06	0.07
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.15	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.39 s
Total hot run time: 29.28 s

hello-stephen · 2025-04-30T05:07:10Z

BE UT Coverage Report

Increment line coverage 100.00% (14/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.80% (14771/26953)
Line Coverage	43.92% (129621/295135)
Region Coverage	42.62% (66159/155229)
Branch Coverage	37.22% (33395/89716)

hello-stephen · 2025-04-30T06:08:21Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	80.18% (21213/26456)
Line Coverage	74.26% (218804/294649)
Region Coverage	72.60% (131866/181633)
Branch Coverage	65.92% (67417/102276)

freemandealer · 2025-04-30T08:17:59Z

run cloud_p0

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer · 2025-05-07T06:28:21Z

run buildall

github-actions · 2025-05-07T07:02:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-07T07:02:36Z

PR approved by anyone and no changes requested.

doris-robot · 2025-05-07T07:05:26Z

TPC-H: Total hot run time: 34534 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17f1dae5f4331c09d47f2cd9deddd791976cc48e, data reload: false
------ Round 1 ----------------------------------
q1	25573	5062	5020	5020
q2	2069	283	188	188
q3	10392	1219	695	695
q4	10656	998	514	514
q5	7525	2420	2384	2384
q6	184	167	132	132
q7	917	780	620	620
q8	9314	1288	1070	1070
q9	6923	5162	5163	5162
q10	6846	2308	1883	1883
q11	472	287	263	263
q12	350	351	209	209
q13	17908	3761	3110	3110
q14	238	230	208	208
q15	531	485	476	476
q16	413	434	389	389
q17	603	858	360	360
q18	7460	7584	7657	7584
q19	3447	976	541	541
q20	338	339	221	221
q21	4134	3595	2480	2480
q22	1086	1050	1025	1025
Total cold run time: 117379 ms
Total hot run time: 34534 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5185	5099	5125	5099
q2	248	333	235	235
q3	2192	2709	2359	2359
q4	1341	1885	1462	1462
q5	4476	4422	4360	4360
q6	215	169	126	126
q7	1978	1920	1740	1740
q8	2576	2641	2518	2518
q9	7214	6972	7123	6972
q10	3052	3143	2714	2714
q11	574	520	482	482
q12	675	735	638	638
q13	3486	3961	3338	3338
q14	294	291	275	275
q15	539	470	465	465
q16	431	492	431	431
q17	1148	1446	1368	1368
q18	7745	7605	7375	7375
q19	827	833	1004	833
q20	1886	1924	1843	1843
q21	5154	4877	4666	4666
q22	1054	1064	1024	1024
Total cold run time: 52290 ms
Total hot run time: 50323 ms

doris-robot · 2025-05-07T07:17:20Z

TPC-DS: Total hot run time: 192804 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17f1dae5f4331c09d47f2cd9deddd791976cc48e, data reload: false
query1	1404	1126	1094	1094
query2	6289	1805	1806	1805
query3	11052	4606	4666	4606
query4	25788	23729	23320	23320
query5	4418	611	448	448
query6	311	238	208	208
query7	4012	496	281	281
query8	323	253	250	250
query9	8458	2569	2595	2569
query10	450	328	277	277
query11	15294	14996	14802	14802
query12	157	110	109	109
query13	1582	522	394	394
query14	8798	6207	6096	6096
query15	208	187	168	168
query16	7569	624	472	472
query17	1198	777	630	630
query18	2077	428	324	324
query19	212	200	175	175
query20	133	130	123	123
query21	215	127	113	113
query22	4608	4470	4457	4457
query23	34329	33477	33472	33472
query24	8429	2421	2437	2421
query25	526	461	410	410
query26	1238	265	151	151
query27	2906	510	340	340
query28	4912	2150	2127	2127
query29	704	559	446	446
query30	288	220	189	189
query31	890	870	802	802
query32	73	65	64	64
query33	532	416	316	316
query34	823	875	532	532
query35	805	813	767	767
query36	957	991	927	927
query37	110	103	77	77
query38	4329	4277	4172	4172
query39	1499	1439	1441	1439
query40	218	130	106	106
query41	54	51	54	51
query42	118	107	108	107
query43	505	505	482	482
query44	1330	808	825	808
query45	183	183	174	174
query46	848	1012	667	667
query47	1792	1861	1817	1817
query48	397	409	319	319
query49	749	489	422	422
query50	654	702	416	416
query51	4340	4219	4252	4219
query52	116	109	101	101
query53	232	271	197	197
query54	620	603	521	521
query55	87	84	81	81
query56	305	303	323	303
query57	1192	1192	1119	1119
query58	287	268	261	261
query59	2619	2761	2649	2649
query60	336	345	308	308
query61	135	128	125	125
query62	783	730	670	670
query63	229	192	193	192
query64	4131	1101	721	721
query65	4487	4383	4343	4343
query66	983	416	298	298
query67	15736	15529	15419	15419
query68	8532	877	519	519
query69	482	302	256	256
query70	1234	1076	1174	1076
query71	463	325	295	295
query72	5915	4797	4952	4797
query73	753	675	354	354
query74	8860	9101	8797	8797
query75	3909	3181	2693	2693
query76	3739	1184	755	755
query77	790	379	284	284
query78	9949	10286	9295	9295
query79	2475	805	564	564
query80	616	501	432	432
query81	476	263	219	219
query82	448	125	96	96
query83	286	250	243	243
query84	307	108	89	89
query85	778	367	315	315
query86	369	302	295	295
query87	4448	4454	4309	4309
query88	3402	2244	2175	2175
query89	388	317	280	280
query90	1849	208	207	207
query91	144	145	111	111
query92	82	64	61	61
query93	1913	983	582	582
query94	662	423	302	302
query95	376	299	292	292
query96	484	566	267	267
query97	3175	3238	3103	3103
query98	227	206	194	194
query99	1422	1408	1278	1278
Total cold run time: 280325 ms
Total hot run time: 192804 ms

doris-robot · 2025-05-07T07:22:30Z

ClickBench: Total hot run time: 29.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17f1dae5f4331c09d47f2cd9deddd791976cc48e, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.18
query4	1.59	0.19	0.19
query5	0.61	0.59	0.58
query6	1.22	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.58	0.52	0.51
query10	0.56	0.56	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	0.79	0.80	0.80
query15	0.87	0.84	0.84
query16	0.38	0.39	0.39
query17	1.03	1.02	1.04
query18	0.21	0.20	0.20
query19	1.85	1.82	1.78
query20	0.02	0.01	0.01
query21	15.41	0.91	0.55
query22	0.77	1.25	0.76
query23	14.73	1.40	0.60
query24	6.64	2.26	0.74
query25	0.50	0.15	0.09
query26	0.67	0.16	0.12
query27	0.05	0.05	0.05
query28	9.31	0.88	0.41
query29	12.54	3.92	3.27
query30	0.26	0.09	0.06
query31	2.82	0.58	0.38
query32	3.23	0.54	0.46
query33	3.01	3.10	3.23
query34	15.75	5.09	4.48
query35	4.50	4.50	4.50
query36	0.66	0.50	0.48
query37	0.08	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.12	0.12
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 102.42 s
Total hot run time: 29.14 s

hello-stephen · 2025-05-07T08:13:54Z

BE UT Coverage Report

Increment line coverage 100.00% (14/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.51% (14845/26743)
Line Coverage	44.54% (130918/293948)
Region Coverage	43.27% (66692/154128)
Branch Coverage	37.88% (33659/88866)

hello-stephen · 2025-05-07T11:29:14Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	58.55% (15363/26241)
Line Coverage	47.81% (140284/293435)
Region Coverage	44.52% (80392/180569)
Branch Coverage	38.08% (38627/101442)

freemandealer · 2025-05-08T07:01:39Z

run buildall

doris-robot · 2025-05-08T08:16:43Z

TPC-H: Total hot run time: 34153 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b06cc05418ad5c134767cf818a4cd8414707d455, data reload: false
------ Round 1 ----------------------------------
q1	26407	4970	5145	4970
q2	2071	269	193	193
q3	10404	1244	722	722
q4	10230	1011	564	564
q5	7542	2412	2412	2412
q6	182	161	134	134
q7	927	765	629	629
q8	9333	1367	1147	1147
q9	6778	5132	5152	5132
q10	6823	2309	1879	1879
q11	481	297	278	278
q12	361	358	223	223
q13	17771	3708	3100	3100
q14	219	229	210	210
q15	519	487	492	487
q16	430	443	378	378
q17	621	881	386	386
q18	7585	7292	7052	7052
q19	1220	964	571	571
q20	343	343	237	237
q21	4108	2664	2489	2489
q22	1025	1015	960	960
Total cold run time: 115380 ms
Total hot run time: 34153 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5068	5074	5081	5074
q2	243	326	231	231
q3	2157	2676	2301	2301
q4	1343	1770	1514	1514
q5	4612	4437	4322	4322
q6	220	168	124	124
q7	1951	1901	1710	1710
q8	2586	2560	2614	2560
q9	7108	7247	7175	7175
q10	3139	3203	2724	2724
q11	584	520	489	489
q12	681	775	597	597
q13	3515	3918	3248	3248
q14	283	305	278	278
q15	530	484	500	484
q16	450	497	440	440
q17	1164	1748	1385	1385
q18	7867	7451	7512	7451
q19	858	863	947	863
q20	1946	2011	1845	1845
q21	4941	4595	4556	4556
q22	1064	1015	994	994
Total cold run time: 52310 ms
Total hot run time: 50365 ms

doris-robot · 2025-05-08T08:28:25Z

TPC-DS: Total hot run time: 186465 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b06cc05418ad5c134767cf818a4cd8414707d455, data reload: false
query1	1007	470	512	470
query2	6565	1817	1761	1761
query3	6743	227	228	227
query4	26593	23286	23404	23286
query5	4368	637	446	446
query6	322	196	190	190
query7	4614	495	303	303
query8	284	234	220	220
query9	8575	2553	2569	2553
query10	491	331	270	270
query11	15529	15134	14871	14871
query12	164	106	102	102
query13	1640	508	390	390
query14	8810	6327	6199	6199
query15	201	191	170	170
query16	7153	657	498	498
query17	1105	758	569	569
query18	1960	400	296	296
query19	192	181	153	153
query20	118	126	121	121
query21	211	129	104	104
query22	4060	4134	4019	4019
query23	33896	32992	33099	32992
query24	8456	2455	2442	2442
query25	539	453	393	393
query26	1236	286	159	159
query27	2740	491	329	329
query28	4369	2084	2079	2079
query29	757	558	458	458
query30	288	219	191	191
query31	922	854	758	758
query32	73	66	72	66
query33	561	383	326	326
query34	805	828	520	520
query35	819	800	737	737
query36	959	991	884	884
query37	111	101	75	75
query38	4112	4180	4109	4109
query39	1456	1415	1414	1414
query40	214	121	109	109
query41	57	54	52	52
query42	122	105	108	105
query43	502	485	475	475
query44	1328	819	808	808
query45	184	173	169	169
query46	860	1033	641	641
query47	1798	1771	1688	1688
query48	370	433	311	311
query49	770	509	422	422
query50	684	689	402	402
query51	4190	4171	4198	4171
query52	112	106	96	96
query53	234	267	195	195
query54	601	581	510	510
query55	85	86	92	86
query56	312	314	289	289
query57	1170	1145	1081	1081
query58	279	255	266	255
query59	2659	2604	2582	2582
query60	343	338	317	317
query61	158	159	149	149
query62	797	733	670	670
query63	239	201	196	196
query64	4455	1124	676	676
query65	4332	4267	4233	4233
query66	1155	404	325	325
query67	15928	15565	15479	15479
query68	8168	890	511	511
query69	476	297	264	264
query70	1163	1069	1055	1055
query71	454	328	292	292
query72	5408	4679	4688	4679
query73	697	575	356	356
query74	9106	9137	8887	8887
query75	3784	3204	2725	2725
query76	3638	1191	749	749
query77	790	385	283	283
query78	10051	9987	9429	9429
query79	6624	772	550	550
query80	718	550	463	463
query81	487	262	221	221
query82	760	130	101	101
query83	294	256	242	242
query84	293	99	86	86
query85	795	362	316	316
query86	395	315	298	298
query87	4443	4465	4325	4325
query88	4047	2264	2288	2264
query89	458	312	278	278
query90	1844	219	218	218
query91	158	151	114	114
query92	80	66	55	55
query93	4202	933	576	576
query94	678	413	301	301
query95	383	301	292	292
query96	476	572	278	278
query97	3187	3279	3133	3133
query98	237	213	210	210
query99	1414	1409	1272	1272
Total cold run time: 282450 ms
Total hot run time: 186465 ms

doris-robot · 2025-05-08T08:33:44Z

ClickBench: Total hot run time: 28.88 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b06cc05418ad5c134767cf818a4cd8414707d455, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.61	0.59	0.59
query6	1.18	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.52	0.52
query10	0.57	0.58	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	0.78	0.80	0.83
query15	0.89	0.87	0.85
query16	0.39	0.40	0.39
query17	1.00	1.01	1.01
query18	0.21	0.19	0.19
query19	1.88	1.81	1.86
query20	0.01	0.01	0.02
query21	15.41	0.91	0.56
query22	0.76	1.29	0.64
query23	14.83	1.38	0.62
query24	7.04	1.43	0.59
query25	0.50	0.19	0.08
query26	0.56	0.17	0.14
query27	0.06	0.05	0.06
query28	9.52	0.88	0.44
query29	12.52	3.96	3.23
query30	0.25	0.11	0.07
query31	2.80	0.60	0.38
query32	3.22	0.54	0.46
query33	3.07	3.03	3.14
query34	15.88	5.12	4.50
query35	4.55	4.48	4.52
query36	0.68	0.51	0.48
query37	0.09	0.06	0.05
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.02	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.19 s
Total hot run time: 28.88 s

hello-stephen · 2025-05-08T10:27:21Z

BE UT Coverage Report

Increment line coverage 100.00% (14/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.67% (14892/26751)
Line Coverage	44.48% (131328/295245)
Region Coverage	43.38% (66167/152524)
Branch Coverage	38.06% (33787/88784)

hello-stephen · 2025-05-08T11:51:36Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.22% (20795/26250)
Line Coverage	72.47% (213618/294748)
Region Coverage	70.64% (126374/178886)
Branch Coverage	64.26% (65109/101316)

freemandealer · 2025-05-13T01:18:34Z

run external

freemandealer · 2025-05-13T01:18:38Z

run p0

freemandealer · 2025-05-13T02:18:51Z

run external

hello-stephen · 2025-05-13T03:43:17Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.40% (20843/26250)
Line Coverage	72.63% (214077/294748)
Region Coverage	70.78% (126623/178886)
Branch Coverage	64.34% (65185/101316)

hello-stephen · 2025-05-13T03:52:22Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.40% (20843/26250)
Line Coverage	72.63% (214077/294748)
Region Coverage	70.78% (126623/178886)
Branch Coverage	64.34% (65185/101316)

dataroaring

LGTM

originally, fd is still hold by FDCache after clear file cache (sync) will cause disk space never reaching zero Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

…0893) Cherry-picked from #50555 Signed-off-by: zhengyu <zhangzhengyu@selectdb.com> Co-authored-by: zhengyu <zhangzhengyu@selectdb.com>

originally, fd is still hold by FDCache after clear file cache (sync) will cause disk space never reaching zero Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer requested review from dataroaring and gavinchou as code owners April 29, 2025 14:15

[fix](cloud) release fd after clear file cache
f4ccae6
originally, fd is still hold by FDCache after clear file cache (sync) will cause disk space never reaching zero Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer force-pushed the fix-clear-cache-fd-unreleased branch from 6fdf686 to f4ccae6Compare April 29, 2025 14:18

gavinchou reviewed Apr 30, 2025
View reviewed changes

add timer
0863ab4
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

to make reviewer happy
17f1dae
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

gavinchou approved these changes May 7, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 7, 2025

github-actionsBot added the reviewed label May 7, 2025

Hastyshell approved these changes May 7, 2025
View reviewed changes

Yukang-Lian approved these changes May 7, 2025
View reviewed changes

gavinchou approved these changes May 8, 2025
View reviewed changes

Merge branch 'master' into fix-clear-cache-fd-unreleased
b06cc05

dataroaring added the dev/3.0.x label May 14, 2025

dataroaring approved these changes May 14, 2025
View reviewed changes

dataroaring merged commit 015c20b into apache:masterMay 14, 2025

github-actionsBot mentioned this pull request May 14, 2025
branch-3.0: [fix](cloud) release fd after clear file cache #50555 #50893
Merged

dataroaring pushed a commit that referenced this pull request May 14, 2025
branch-3.0: [fix](cloud) release fd after clear file cache#50555(#5…
858aaf0
…0893) Cherry-picked from #50555 Signed-off-by: zhengyu <zhangzhengyu@selectdb.com> Co-authored-by: zhengyu <zhangzhengyu@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 14, 2025

gavinchou mentioned this pull request Jun 11, 2025
Release Note 3.0.6 #51585
Closed

Uh oh!

Conversation

freemandealer commented Apr 29, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 29, 2025

Uh oh!

freemandealer commented Apr 29, 2025

Uh oh!

doris-robot commented Apr 29, 2025

Uh oh!

doris-robot commented Apr 29, 2025

Uh oh!

doris-robot commented Apr 29, 2025

Uh oh!

doris-robot commented Apr 29, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 29, 2025

BE Regression P0 && UT Coverage Report

Uh oh!

gavinchouApr 30, 2025

Choose a reason for hiding this comment

Uh oh!

freemandealerApr 30, 2025

Choose a reason for hiding this comment

Uh oh!

gavinchouMay 4, 2025

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Apr 30, 2025

Uh oh!

doris-robot commented Apr 30, 2025

Uh oh!

doris-robot commented Apr 30, 2025

Uh oh!

doris-robot commented Apr 30, 2025

Uh oh!

hello-stephen commented Apr 30, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 30, 2025

BE Regression && UT Coverage Report

Uh oh!

freemandealer commented Apr 30, 2025

Uh oh!

freemandealer commented May 7, 2025

Uh oh!

github-actionsBot commented May 7, 2025

Uh oh!

github-actionsBot commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

hello-stephen commented May 7, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 7, 2025

BE Regression && UT Coverage Report

Uh oh!

freemandealer commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

hello-stephen commented May 8, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 8, 2025

BE Regression && UT Coverage Report