[feat](compaction) Make base compaction output rowset write filecache more adaptive by swjtu-zhanglei · Pull Request #54694 · apache/doris

swjtu-zhanglei · 2025-08-13T10:51:39Z

when base compaction input rowsets (total_cache_size / total_data_size) > file_cache_keep_base_compaction_output_min_hit_ratio
output rowsets will write to filecache

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

swjtu-zhanglei · 2025-08-13T10:51:48Z

run buildall

Thearas · 2025-08-13T10:52:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

swjtu-zhanglei · 2025-08-14T09:37:25Z

run buildall

doris-robot · 2025-08-14T11:08:22Z

TPC-H: Total hot run time: 33793 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f29df92b87381b509e62f23af4bffc8a3883eca, data reload: false
------ Round 1 ----------------------------------
q1	17597	5249	5048	5048
q2	1922	288	182	182
q3	10305	1276	731	731
q4	10260	1040	504	504
q5	8526	2368	2309	2309
q6	203	162	132	132
q7	916	748	598	598
q8	9315	1283	1123	1123
q9	7097	5111	5165	5111
q10	6971	2361	1974	1974
q11	477	287	265	265
q12	358	349	214	214
q13	17782	3640	3038	3038
q14	228	236	215	215
q15	561	480	492	480
q16	419	415	377	377
q17	589	869	359	359
q18	7655	7037	7031	7031
q19	1553	950	578	578
q20	341	328	221	221
q21	3663	3172	2308	2308
q22	1080	1031	995	995
Total cold run time: 107818 ms
Total hot run time: 33793 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5196	5084	5138	5084
q2	246	318	223	223
q3	2125	2670	2304	2304
q4	1362	1780	1389	1389
q5	4253	4501	4534	4501
q6	225	170	129	129
q7	1999	1993	1794	1794
q8	2707	2581	2590	2581
q9	7325	7303	7338	7303
q10	3072	3320	2869	2869
q11	591	511	496	496
q12	701	767	616	616
q13	3503	3896	3358	3358
q14	299	300	292	292
q15	519	491	459	459
q16	476	507	438	438
q17	1284	1557	1375	1375
q18	7925	7793	7228	7228
q19	765	766	849	766
q20	1895	1963	1804	1804
q21	4717	4332	4274	4274
q22	1072	1028	994	994
Total cold run time: 52257 ms
Total hot run time: 50277 ms

doris-robot · 2025-08-14T11:19:54Z

TPC-DS: Total hot run time: 184937 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f29df92b87381b509e62f23af4bffc8a3883eca, data reload: false
query1	972	380	435	380
query2	6511	1721	1736	1721
query3	6750	220	212	212
query4	26427	23335	23283	23283
query5	4345	622	473	473
query6	312	205	204	204
query7	4634	499	285	285
query8	299	226	218	218
query9	8616	2838	2828	2828
query10	491	339	280	280
query11	15388	14975	14905	14905
query12	165	119	117	117
query13	1671	543	412	412
query14	8539	5763	5693	5693
query15	206	182	164	164
query16	7329	663	501	501
query17	1210	728	597	597
query18	1994	425	323	323
query19	193	188	172	172
query20	134	117	112	112
query21	210	122	108	108
query22	4117	4097	4056	4056
query23	34173	33372	33466	33372
query24	8109	2342	2292	2292
query25	550	457	457	457
query26	1231	265	158	158
query27	2766	497	346	346
query28	4296	2205	2183	2183
query29	776	566	451	451
query30	294	233	183	183
query31	869	810	708	708
query32	81	78	73	73
query33	571	365	347	347
query34	785	830	515	515
query35	763	810	720	720
query36	956	994	927	927
query37	121	114	82	82
query38	4151	4075	3942	3942
query39	1463	1423	1407	1407
query40	224	125	115	115
query41	57	57	57	57
query42	125	109	109	109
query43	496	486	465	465
query44	1315	837	826	826
query45	172	170	162	162
query46	844	999	635	635
query47	1767	1791	1698	1698
query48	375	434	306	306
query49	734	473	404	404
query50	632	675	403	403
query51	4043	4100	4072	4072
query52	117	108	105	105
query53	230	259	193	193
query54	585	579	527	527
query55	81	82	89	82
query56	310	308	294	294
query57	1171	1204	1141	1141
query58	275	271	283	271
query59	2653	2767	2555	2555
query60	343	332	328	328
query61	126	120	127	120
query62	803	730	663	663
query63	227	187	186	186
query64	4329	1056	690	690
query65	4317	4183	4187	4183
query66	1166	415	327	327
query67	15421	15098	15016	15016
query68	8081	907	566	566
query69	477	327	281	281
query70	1244	1101	1144	1101
query71	450	329	295	295
query72	5600	4865	5093	4865
query73	791	703	348	348
query74	9238	9110	8867	8867
query75	3716	3075	2638	2638
query76	3631	1108	724	724
query77	777	384	317	317
query78	9603	9621	8844	8844
query79	3924	854	578	578
query80	656	540	477	477
query81	465	254	226	226
query82	545	142	105	105
query83	291	259	241	241
query84	292	106	85	85
query85	783	368	336	336
query86	337	303	303	303
query87	4257	4232	4246	4232
query88	2991	2154	2170	2154
query89	448	316	288	288
query90	1944	219	212	212
query91	141	144	114	114
query92	81	69	68	68
query93	2052	984	627	627
query94	672	396	298	298
query95	397	307	299	299
query96	482	580	271	271
query97	2663	2679	2537	2537
query98	278	215	218	215
query99	1439	1405	1285	1285
Total cold run time: 274152 ms
Total hot run time: 184937 ms

doris-robot · 2025-08-14T11:25:21Z

ClickBench: Total hot run time: 32.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f29df92b87381b509e62f23af4bffc8a3883eca, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.11
query5	0.42	0.42	0.40
query6	1.18	0.64	0.65
query7	0.02	0.01	0.01
query8	0.05	0.03	0.04
query9	0.60	0.52	0.52
query10	0.58	0.58	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.64	0.61	0.61
query14	0.79	0.84	0.82
query15	0.88	0.85	0.84
query16	0.39	0.41	0.40
query17	1.03	1.06	1.07
query18	0.21	0.20	0.20
query19	1.90	1.80	1.82
query20	0.01	0.01	0.00
query21	15.39	0.91	0.56
query22	0.76	1.07	0.71
query23	15.04	1.37	0.64
query24	6.01	1.99	0.61
query25	0.52	0.20	0.07
query26	0.55	0.17	0.13
query27	0.06	0.05	0.05
query28	9.60	0.96	0.42
query29	12.62	3.95	3.26
query30	3.10	3.00	2.94
query31	2.83	0.57	0.39
query32	3.22	0.57	0.47
query33	3.05	3.04	3.12
query34	15.99	5.44	4.86
query35	4.88	4.86	4.92
query36	0.67	0.51	0.49
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.15
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.79 s
Total hot run time: 32.22 s

hello-stephen · 2025-08-14T12:37:11Z

BE UT Coverage Report

Increment line coverage 0.00% (0/82) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.48% (16784/28220)
Line Coverage	48.41% (152607/315223)
Region Coverage	37.40% (116074/310347)
Branch Coverage	40.30% (51195/127031)

hello-stephen · 2025-08-14T13:47:05Z

BE Regression && UT Coverage Report

Increment line coverage 92.68% (76/82) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.92% (22652/27650)
Line Coverage	74.64% (235102/314997)
Region Coverage	61.93% (195459/315588)
Branch Coverage	66.07% (84550/127964)

swjtu-zhanglei · 2025-08-29T01:59:01Z

run buildall

doris-robot · 2025-08-29T03:32:08Z

TPC-H: Total hot run time: 34373 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02e3d20c227cc73757292544cadc2e5d42d16295, data reload: false
------ Round 1 ----------------------------------
q1	17600	5261	5087	5087
q2	2019	318	216	216
q3	10260	1294	739	739
q4	10240	1034	530	530
q5	7562	2403	2398	2398
q6	183	168	138	138
q7	930	775	644	644
q8	9362	1513	1153	1153
q9	6985	5188	5148	5148
q10	7005	2377	2009	2009
q11	491	297	284	284
q12	386	364	230	230
q13	17799	3663	3050	3050
q14	238	241	220	220
q15	564	493	521	493
q16	434	438	397	397
q17	599	870	367	367
q18	7327	7154	7149	7149
q19	1097	965	599	599
q20	347	344	235	235
q21	3829	3151	2318	2318
q22	1060	1036	969	969
Total cold run time: 106317 ms
Total hot run time: 34373 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5101	5091	5126	5091
q2	256	344	232	232
q3	2204	2697	2327	2327
q4	1368	1769	1316	1316
q5	4258	4462	4659	4462
q6	212	180	132	132
q7	2085	1928	1923	1923
q8	2707	2551	2545	2545
q9	7435	7406	7473	7406
q10	3135	3298	2864	2864
q11	569	524	533	524
q12	687	783	672	672
q13	3563	3991	3520	3520
q14	299	296	275	275
q15	522	490	483	483
q16	448	506	458	458
q17	1167	1505	1447	1447
q18	7908	7623	7495	7495
q19	870	849	957	849
q20	1978	2216	1893	1893
q21	4940	4421	4277	4277
q22	1081	1069	991	991
Total cold run time: 52793 ms
Total hot run time: 51182 ms

doris-robot · 2025-08-29T03:43:48Z

TPC-DS: Total hot run time: 186405 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02e3d20c227cc73757292544cadc2e5d42d16295, data reload: false
query1	1046	457	431	431
query2	6579	1802	1795	1795
query3	6760	226	230	226
query4	26319	23813	22915	22915
query5	4403	680	520	520
query6	347	246	233	233
query7	4645	521	297	297
query8	313	267	280	267
query9	8646	2929	2918	2918
query10	488	357	309	309
query11	16152	15317	14777	14777
query12	181	123	134	123
query13	1687	580	462	462
query14	9654	5902	5856	5856
query15	234	193	178	178
query16	7677	668	498	498
query17	1248	760	634	634
query18	2062	441	405	405
query19	197	197	189	189
query20	142	131	118	118
query21	215	128	112	112
query22	4029	4217	4048	4048
query23	33988	32937	33194	32937
query24	8123	2363	2428	2363
query25	585	534	450	450
query26	1240	280	172	172
query27	2727	516	353	353
query28	4439	2265	2253	2253
query29	759	609	506	506
query30	294	224	202	202
query31	928	817	693	693
query32	94	82	84	82
query33	567	403	384	384
query34	809	873	518	518
query35	836	847	774	774
query36	981	1024	924	924
query37	124	115	97	97
query38	4116	4074	3989	3989
query39	1521	1429	1420	1420
query40	231	135	131	131
query41	68	61	61	61
query42	129	116	116	116
query43	530	515	472	472
query44	1367	857	872	857
query45	184	179	170	170
query46	866	1008	639	639
query47	1816	1820	1768	1768
query48	383	426	339	339
query49	776	519	412	412
query50	660	687	395	395
query51	4106	4234	4056	4056
query52	123	118	103	103
query53	256	258	198	198
query54	618	599	541	541
query55	97	95	93	93
query56	344	341	316	316
query57	1196	1199	1120	1120
query58	290	278	280	278
query59	2629	2710	2612	2612
query60	359	355	334	334
query61	195	161	157	157
query62	813	753	653	653
query63	234	197	192	192
query64	4415	1139	826	826
query65	4310	4189	4230	4189
query66	1098	444	337	337
query67	15398	15345	15145	15145
query68	8787	934	594	594
query69	479	332	290	290
query70	1208	1083	1162	1083
query71	465	347	336	336
query72	5961	5128	5354	5128
query73	786	709	364	364
query74	9262	8814	8771	8771
query75	4134	3094	2618	2618
query76	3652	1167	743	743
query77	818	403	332	332
query78	9603	9839	8885	8885
query79	2089	838	585	585
query80	671	589	514	514
query81	476	261	235	235
query82	435	143	114	114
query83	264	262	249	249
query84	255	123	99	99
query85	912	513	425	425
query86	348	306	296	296
query87	4238	4280	4142	4142
query88	3153	2210	2191	2191
query89	395	329	299	299
query90	1953	228	221	221
query91	163	161	136	136
query92	88	82	75	75
query93	1150	993	661	661
query94	717	406	325	325
query95	410	345	327	327
query96	495	570	274	274
query97	2649	2691	2621	2621
query98	243	217	217	217
query99	1464	1417	1317	1317
Total cold run time: 276369 ms
Total hot run time: 186405 ms

doris-robot · 2025-08-29T03:49:28Z

ClickBench: Total hot run time: 32.73 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02e3d20c227cc73757292544cadc2e5d42d16295, data reload: false
query1	0.05	0.05	0.05
query2	0.09	0.06	0.06
query3	0.25	0.08	0.09
query4	1.60	0.12	0.12
query5	0.43	0.44	0.42
query6	1.18	0.66	0.66
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.60	0.53	0.53
query10	0.58	0.58	0.58
query11	0.17	0.12	0.12
query12	0.16	0.12	0.13
query13	0.63	0.63	0.62
query14	0.79	0.85	0.83
query15	0.89	0.85	0.87
query16	0.39	0.42	0.39
query17	1.08	1.06	1.06
query18	0.22	0.20	0.20
query19	1.90	1.86	1.84
query20	0.02	0.01	0.01
query21	15.41	0.96	0.62
query22	0.79	1.23	0.76
query23	14.93	1.42	0.66
query24	7.07	1.14	0.49
query25	0.50	0.17	0.10
query26	0.49	0.15	0.13
query27	0.05	0.05	0.06
query28	9.79	0.93	0.43
query29	12.66	3.97	3.27
query30	3.14	3.10	2.94
query31	2.84	0.60	0.39
query32	3.26	0.57	0.48
query33	3.15	3.10	3.10
query34	16.11	5.48	4.87
query35	4.98	4.96	4.96
query36	0.70	0.50	0.49
query37	0.10	0.07	0.07
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.19	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 107.54 s
Total hot run time: 32.73 s

doris-robot · 2025-08-29T05:08:36Z

BE UT Coverage Report

Increment line coverage 0.00% (0/82) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	51.78% (17150/33120)
Line Coverage	37.26% (156282/419465)
Region Coverage	31.94% (119108/372914)
Branch Coverage	33.25% (52364/157508)

swjtu-zhanglei · 2025-08-29T08:04:12Z

run buildall

swjtu-zhanglei · 2025-08-29T08:05:07Z

run buildall

doris-robot · 2025-08-29T09:28:23Z

TPC-H: Total hot run time: 34116 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4cb44e0cfa5ddf619eae1b47f8a8746bd7d517e, data reload: false
------ Round 1 ----------------------------------
q1	17615	5159	5006	5006
q2	1992	350	217	217
q3	10239	1290	701	701
q4	10242	1044	541	541
q5	7503	2456	2359	2359
q6	179	168	137	137
q7	963	754	634	634
q8	9345	1424	1138	1138
q9	7152	5149	5206	5149
q10	6888	2415	1990	1990
q11	526	303	305	303
q12	351	358	222	222
q13	17778	3651	3062	3062
q14	243	256	224	224
q15	557	497	490	490
q16	442	436	378	378
q17	592	886	360	360
q18	7585	7220	7105	7105
q19	1087	973	558	558
q20	337	345	235	235
q21	3805	3219	2342	2342
q22	1071	1039	965	965
Total cold run time: 106492 ms
Total hot run time: 34116 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5120	5199	5094	5094
q2	247	345	232	232
q3	2174	2706	2331	2331
q4	1351	1790	1304	1304
q5	4207	4499	4647	4499
q6	230	194	133	133
q7	2043	1968	1863	1863
q8	2668	2585	2683	2585
q9	7607	7555	7277	7277
q10	3144	3318	2843	2843
q11	597	527	508	508
q12	703	769	656	656
q13	3561	3864	3473	3473
q14	296	301	282	282
q15	520	480	498	480
q16	453	505	458	458
q17	1136	1597	1381	1381
q18	8055	7831	7706	7706
q19	866	832	839	832
q20	2038	2205	1888	1888
q21	4981	4672	4704	4672
q22	1160	1113	1015	1015
Total cold run time: 53157 ms
Total hot run time: 51512 ms

doris-robot · 2025-08-29T09:40:01Z

TPC-DS: Total hot run time: 184892 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4cb44e0cfa5ddf619eae1b47f8a8746bd7d517e, data reload: false
query1	1087	437	444	437
query2	6559	1790	1760	1760
query3	6761	234	236	234
query4	26317	23230	23153	23153
query5	4342	641	532	532
query6	346	252	225	225
query7	4650	518	295	295
query8	303	253	243	243
query9	8618	2877	2897	2877
query10	478	335	293	293
query11	15864	15315	14757	14757
query12	180	122	121	121
query13	1686	554	438	438
query14	9665	5791	5868	5791
query15	232	196	195	195
query16	7673	675	507	507
query17	1264	794	657	657
query18	2059	457	357	357
query19	216	205	190	190
query20	135	130	130	130
query21	226	133	113	113
query22	4351	4184	4186	4184
query23	34049	33068	32908	32908
query24	8144	2432	2403	2403
query25	574	534	443	443
query26	1244	280	168	168
query27	2736	523	353	353
query28	4385	2293	2262	2262
query29	758	627	502	502
query30	290	244	208	208
query31	918	839	726	726
query32	94	82	87	82
query33	571	415	353	353
query34	830	862	537	537
query35	827	850	767	767
query36	993	1009	918	918
query37	129	116	126	116
query38	4040	4009	4013	4009
query39	1502	1444	1425	1425
query40	230	134	135	134
query41	65	66	62	62
query42	130	113	116	113
query43	502	510	494	494
query44	1353	868	868	868
query45	188	175	173	173
query46	854	1037	665	665
query47	1798	1808	1760	1760
query48	402	431	329	329
query49	786	528	443	443
query50	642	702	406	406
query51	4122	4195	4370	4195
query52	122	114	105	105
query53	254	264	206	206
query54	626	620	552	552
query55	96	92	94	92
query56	348	330	325	325
query57	1201	1215	1142	1142
query58	294	279	297	279
query59	2660	2837	2619	2619
query60	381	356	360	356
query61	165	159	162	159
query62	827	769	688	688
query63	256	205	194	194
query64	4402	1148	965	965
query65	4338	4224	4239	4224
query66	1116	488	379	379
query67	15583	15343	15108	15108
query68	7873	942	597	597
query69	487	346	307	307
query70	1195	1171	1161	1161
query71	483	420	328	328
query72	5778	4907	2717	2717
query73	665	598	359	359
query74	8952	9151	8695	8695
query75	3803	3087	2625	2625
query76	3616	1140	749	749
query77	825	422	350	350
query78	9679	9837	8843	8843
query79	2466	878	596	596
query80	653	582	514	514
query81	503	265	230	230
query82	483	150	111	111
query83	264	267	252	252
query84	269	120	90	90
query85	894	461	444	444
query86	396	340	315	315
query87	4299	4305	4164	4164
query88	3669	2263	2261	2261
query89	406	341	296	296
query90	1867	232	235	232
query91	168	168	132	132
query92	97	75	76	75
query93	1908	990	663	663
query94	693	428	327	327
query95	414	335	325	325
query96	500	587	283	283
query97	2708	2721	2640	2640
query98	266	219	214	214
query99	1471	1466	1293	1293
Total cold run time: 276587 ms
Total hot run time: 184892 ms

doris-robot · 2025-08-29T09:45:41Z

ClickBench: Total hot run time: 33.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4cb44e0cfa5ddf619eae1b47f8a8746bd7d517e, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.46	0.44	0.43
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.60	0.53	0.54
query10	0.59	0.59	0.58
query11	0.17	0.11	0.12
query12	0.15	0.13	0.12
query13	0.62	0.62	0.63
query14	0.79	0.85	0.85
query15	0.87	0.84	0.85
query16	0.39	0.40	0.39
query17	1.02	1.09	1.08
query18	0.22	0.20	0.20
query19	1.93	1.93	1.79
query20	0.02	0.01	0.01
query21	15.42	0.96	0.58
query22	0.76	1.18	0.73
query23	14.86	1.36	0.60
query24	6.12	0.87	1.54
query25	0.46	0.30	0.22
query26	0.63	0.16	0.13
query27	0.05	0.05	0.06
query28	10.13	0.93	0.43
query29	12.57	3.93	3.25
query30	3.12	3.01	3.01
query31	2.83	0.59	0.39
query32	3.24	0.54	0.47
query33	3.10	3.10	3.24
query34	16.12	5.43	4.90
query35	4.96	4.86	4.90
query36	0.69	0.53	0.50
query37	0.10	0.08	0.08
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.19	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 106.72 s
Total hot run time: 33.06 s

doris-robot · 2025-09-10T10:51:06Z

TPC-DS: Total hot run time: 189456 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e61b42460c52970c48de639a66ece148ab1822a6, data reload: false
query1	1066	439	439	439
query2	6578	1717	1719	1717
query3	6754	231	228	228
query4	26794	24010	22923	22923
query5	5265	662	515	515
query6	354	240	230	230
query7	4667	527	316	316
query8	315	287	278	278
query9	8653	2968	2996	2968
query10	566	359	312	312
query11	16088	15073	14710	14710
query12	183	129	119	119
query13	1694	562	460	460
query14	10953	9497	9402	9402
query15	210	197	182	182
query16	7727	686	498	498
query17	1602	830	668	668
query18	2069	455	411	411
query19	206	204	184	184
query20	137	129	124	124
query21	219	132	123	123
query22	4017	4146	4053	4053
query23	34308	32995	33025	32995
query24	8272	2417	2408	2408
query25	571	518	441	441
query26	1228	279	172	172
query27	2688	514	373	373
query28	4337	2291	2253	2253
query29	800	616	500	500
query30	329	234	203	203
query31	889	816	740	740
query32	97	88	88	88
query33	582	411	353	353
query34	812	887	537	537
query35	841	843	780	780
query36	996	1049	909	909
query37	129	115	99	99
query38	3495	3529	3524	3524
query39	1491	1431	1417	1417
query40	224	138	129	129
query41	66	62	64	62
query42	133	130	125	125
query43	504	543	514	514
query44	1387	878	863	863
query45	185	178	173	173
query46	861	1024	667	667
query47	1797	1816	1739	1739
query48	409	439	321	321
query49	744	509	413	413
query50	665	695	412	412
query51	3917	3960	3899	3899
query52	128	125	115	115
query53	258	287	217	217
query54	637	613	564	564
query55	98	93	89	89
query56	351	342	332	332
query57	1167	1213	1129	1129
query58	295	288	283	283
query59	2632	2663	2540	2540
query60	369	364	361	361
query61	166	173	164	164
query62	797	742	689	689
query63	235	201	208	201
query64	4441	1187	891	891
query65	4061	4009	3989	3989
query66	1110	437	370	370
query67	15475	15124	14998	14998
query68	8978	935	587	587
query69	507	350	294	294
query70	1366	1214	1325	1214
query71	569	363	339	339
query72	5999	5034	5111	5034
query73	762	638	362	362
query74	9101	9349	8894	8894
query75	4447	3278	2778	2778
query76	3754	1211	769	769
query77	823	421	346	346
query78	9674	9810	8836	8836
query79	2212	816	597	597
query80	670	670	528	528
query81	503	265	227	227
query82	475	162	138	138
query83	264	261	253	253
query84	263	109	90	90
query85	867	474	432	432
query86	401	323	297	297
query87	3746	3801	3672	3672
query88	3429	2192	2191	2191
query89	399	333	294	294
query90	1932	232	228	228
query91	161	163	133	133
query92	91	78	72	72
query93	1660	997	656	656
query94	689	429	342	342
query95	417	336	339	336
query96	496	575	279	279
query97	2928	2995	2834	2834
query98	270	225	219	219
query99	1381	1385	1284	1284
Total cold run time: 280328 ms
Total hot run time: 189456 ms

doris-robot · 2025-09-10T10:56:35Z

ClickBench: Total hot run time: 29.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e61b42460c52970c48de639a66ece148ab1822a6, data reload: false
query1	0.06	0.05	0.05
query2	0.09	0.06	0.05
query3	0.24	0.08	0.08
query4	1.60	0.11	0.12
query5	0.28	0.29	0.28
query6	1.18	0.65	0.64
query7	0.03	0.02	0.02
query8	0.06	0.04	0.05
query9	0.63	0.53	0.53
query10	0.59	0.59	0.58
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.64	0.62
query14	1.01	1.03	1.02
query15	0.85	0.87	0.86
query16	0.40	0.40	0.40
query17	1.03	1.05	1.03
query18	0.21	0.20	0.20
query19	1.92	1.80	1.81
query20	0.01	0.02	0.01
query21	15.40	0.97	0.59
query22	0.82	1.33	0.70
query23	14.75	1.38	0.62
query24	6.92	1.73	0.59
query25	0.50	0.27	0.13
query26	0.64	0.16	0.12
query27	0.05	0.05	0.06
query28	9.71	0.91	0.43
query29	12.55	3.92	3.24
query30	0.30	0.13	0.12
query31	2.84	0.59	0.38
query32	3.25	0.56	0.47
query33	3.09	3.14	3.12
query34	16.04	5.43	4.85
query35	4.86	4.87	4.90
query36	0.69	0.51	0.50
query37	0.11	0.08	0.07
query38	0.07	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.11 s
Total hot run time: 29.71 s

doris-robot · 2025-09-10T12:26:00Z

BE UT Coverage Report

Increment line coverage 32.14% (27/84) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	51.96% (17303/33299)
Line Coverage	37.30% (157652/422676)
Region Coverage	31.95% (120327/376559)
Branch Coverage	33.32% (52824/158523)

github-actions · 2025-09-10T12:55:30Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2025-09-10T13:20:11Z

BE Regression && UT Coverage Report

Increment line coverage 92.86% (78/84) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.72% (23124/32698)
Line Coverage	57.12% (241304/422422)
Region Coverage	52.49% (200502/381988)
Branch Coverage	54.11% (86265/159426)

… more adaptive * when base compaction input rowsets (total_cache_size / total_data_size) > file_cache_keep_base_compaction_output_min_hit_ratio output rowsets will write to filecache

swjtu-zhanglei · 2025-09-11T11:19:05Z

run buildall

doris-robot · 2025-09-11T12:42:32Z

TPC-H: Total hot run time: 36788 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0479fb74f975981fbca428770b5968165c30815, data reload: false
------ Round 1 ----------------------------------
q1	17611	5497	5343	5343
q2	2017	335	215	215
q3	10256	1455	753	753
q4	10222	1150	572	572
q5	7550	2773	2767	2767
q6	215	182	141	141
q7	1046	834	666	666
q8	9366	1585	1358	1358
q9	7031	5273	5262	5262
q10	6993	2501	2021	2021
q11	512	308	288	288
q12	369	442	236	236
q13	17762	3855	3079	3079
q14	257	244	224	224
q15	574	497	510	497
q16	1021	1019	961	961
q17	618	932	372	372
q18	7969	7181	7067	7067
q19	1226	1096	601	601
q20	380	364	241	241
q21	4327	3475	3150	3150
q22	1130	1057	974	974
Total cold run time: 108452 ms
Total hot run time: 36788 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5467	5481	5485	5481
q2	260	359	230	230
q3	2341	2825	2393	2393
q4	1419	2067	1667	1667
q5	4840	4627	4721	4627
q6	260	181	134	134
q7	2246	2105	1986	1986
q8	3041	2912	3206	2912
q9	7260	7219	7243	7219
q10	3418	3472	3013	3013
q11	656	518	529	518
q12	735	795	625	625
q13	3779	4112	3347	3347
q14	317	327	294	294
q15	578	527	492	492
q16	1069	1123	1137	1123
q17	1300	1767	1525	1525
q18	7962	7762	7763	7762
q19	874	889	1125	889
q20	2131	2096	1933	1933
q21	5004	4555	4494	4494
q22	1142	1046	996	996
Total cold run time: 56099 ms
Total hot run time: 53660 ms

doris-robot · 2025-09-11T12:54:20Z

TPC-DS: Total hot run time: 188863 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0479fb74f975981fbca428770b5968165c30815, data reload: false
query1	1058	456	431	431
query2	6562	1707	1719	1707
query3	6751	226	227	226
query4	26754	23281	23005	23005
query5	4855	644	535	535
query6	350	247	220	220
query7	4669	508	296	296
query8	296	259	278	259
query9	8655	2897	2902	2897
query10	500	339	301	301
query11	15538	15092	14799	14799
query12	173	121	113	113
query13	1675	542	430	430
query14	10863	9186	9213	9186
query15	212	199	187	187
query16	7735	662	497	497
query17	1216	774	656	656
query18	2062	464	351	351
query19	228	209	196	196
query20	137	136	127	127
query21	219	132	121	121
query22	4002	4339	4076	4076
query23	34038	33080	32951	32951
query24	8226	2394	2402	2394
query25	634	585	452	452
query26	1236	280	168	168
query27	2718	515	357	357
query28	4413	2271	2232	2232
query29	782	620	495	495
query30	291	241	198	198
query31	933	794	711	711
query32	93	80	80	80
query33	577	408	350	350
query34	851	857	534	534
query35	836	825	753	753
query36	949	1027	917	917
query37	124	122	95	95
query38	3528	3535	3551	3535
query39	1490	1418	1433	1418
query40	220	143	128	128
query41	69	63	59	59
query42	128	120	117	117
query43	516	496	478	478
query44	1329	889	886	886
query45	196	175	177	175
query46	856	1009	646	646
query47	1777	1810	1815	1810
query48	390	416	329	329
query49	764	511	414	414
query50	669	690	414	414
query51	3906	3926	3887	3887
query52	118	121	109	109
query53	255	264	198	198
query54	611	598	551	551
query55	99	94	90	90
query56	336	342	321	321
query57	1226	1199	1143	1143
query58	300	290	289	289
query59	2529	2649	2524	2524
query60	365	347	346	346
query61	168	154	166	154
query62	850	707	706	706
query63	235	196	196	196
query64	4431	1189	820	820
query65	4063	3967	3933	3933
query66	1105	460	366	366
query67	15720	15330	15051	15051
query68	8772	927	586	586
query69	492	332	300	300
query70	1366	1251	1290	1251
query71	585	344	325	325
query72	5843	5043	5118	5043
query73	746	605	366	366
query74	9116	9174	8824	8824
query75	4101	3244	2804	2804
query76	3703	1151	729	729
query77	814	410	336	336
query78	9514	9603	8848	8848
query79	4934	843	567	567
query80	695	578	528	528
query81	464	257	225	225
query82	606	165	139	139
query83	297	264	257	257
query84	300	113	104	104
query85	884	474	429	429
query86	348	323	302	302
query87	3762	3797	3604	3604
query88	3038	2189	2194	2189
query89	454	324	304	304
query90	1937	224	225	224
query91	164	170	140	140
query92	87	76	73	73
query93	2299	975	648	648
query94	702	436	324	324
query95	404	332	335	332
query96	478	577	283	283
query97	2989	2987	2874	2874
query98	240	215	217	215
query99	1449	1401	1294	1294
Total cold run time: 281043 ms
Total hot run time: 188863 ms

doris-robot · 2025-09-11T12:59:49Z

ClickBench: Total hot run time: 29.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0479fb74f975981fbca428770b5968165c30815, data reload: false
query1	0.06	0.05	0.05
query2	0.09	0.06	0.06
query3	0.25	0.08	0.09
query4	1.61	0.12	0.12
query5	0.28	0.27	0.26
query6	1.20	0.65	0.64
query7	0.04	0.03	0.02
query8	0.06	0.04	0.04
query9	0.63	0.52	0.53
query10	0.57	0.59	0.58
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.63	0.63	0.62
query14	1.04	1.04	1.02
query15	0.88	0.86	0.84
query16	0.40	0.40	0.40
query17	1.06	1.06	1.07
query18	0.21	0.21	0.20
query19	1.90	1.82	1.80
query20	0.01	0.02	0.01
query21	15.52	0.97	0.57
query22	0.76	1.17	0.69
query23	14.97	1.38	0.61
query24	7.19	1.24	0.72
query25	0.46	0.26	0.10
query26	0.54	0.16	0.14
query27	0.06	0.06	0.06
query28	9.28	0.90	0.43
query29	12.57	3.85	3.26
query30	0.28	0.12	0.11
query31	2.84	0.59	0.38
query32	3.26	0.56	0.48
query33	3.12	3.06	3.16
query34	16.20	5.50	4.86
query35	4.94	4.96	4.91
query36	0.69	0.51	0.50
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.47 s
Total hot run time: 29.8 s

hello-stephen · 2025-09-11T14:16:51Z

BE UT Coverage Report

Increment line coverage 32.14% (27/84) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.06% (17317/33266)
Line Coverage	37.36% (157903/422600)
Region Coverage	31.97% (120555/377119)
Branch Coverage	33.35% (52904/158653)

hello-stephen · 2025-09-11T15:23:17Z

BE Regression && UT Coverage Report

Increment line coverage 92.86% (78/84) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.73% (23085/32637)
Line Coverage	57.12% (241149/422144)
Region Coverage	52.62% (201252/382430)
Branch Coverage	54.15% (86363/159492)

dataroaring

LGTM

… more adaptive (apache#54694) * when base compaction input rowsets (total_cache_size / total_data_size) > file_cache_keep_base_compaction_output_min_hit_ratio output rowsets will write to filecache

…te filecache more adaptive (apache#54694) Cherry-picked from apache#54694 * when base compaction input rowsets (total_cache_size / total_data_size) > file_cache_keep_base_compaction_output_min_hit_ratio output rowsets will write to filecache

…te filecache more adaptive #54694 (#56278) Cherry-picked from #54694

swjtu-zhanglei changed the title ~~save workspace~~save workspace for debug testAug 13, 2025

swjtu-zhanglei force-pushed the master-fix13 branch from 0113e6e to 6f29df9Compare August 14, 2025 09:35

swjtu-zhanglei force-pushed the master-fix13 branch from 6f29df9 to 02e3d20Compare August 29, 2025 01:56

swjtu-zhanglei changed the title ~~save workspace for debug test~~[feat](compaction) Make base compaction output rowset write filecache more adaptiveAug 29, 2025

swjtu-zhanglei marked this pull request as ready for review August 29, 2025 01:58

swjtu-zhanglei requested review from dataroaring and gavinchou as code owners August 29, 2025 01:58

swjtu-zhanglei force-pushed the master-fix13 branch from 02e3d20 to 5088acbCompare August 29, 2025 02:16

swjtu-zhanglei force-pushed the master-fix13 branch 3 times, most recently from 16412c3 to 6689b6dCompare August 29, 2025 08:03

swjtu-zhanglei force-pushed the master-fix13 branch from 6689b6d to e4cb44eCompare August 29, 2025 08:04

gavinchou previously approved these changes Sep 10, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 10, 2025

wyxxxcat approved these changes Sep 10, 2025
View reviewed changes

gavinchou added the dev/3.1.x label Sep 10, 2025

[feat](compaction) Make base compaction output rowset write filecache…
a0479fb
… more adaptive * when base compaction input rowsets (total_cache_size / total_data_size) > file_cache_keep_base_compaction_output_min_hit_ratio output rowsets will write to filecache

swjtu-zhanglei dismissed gavinchou’s stale review via a0479fbSeptember 11, 2025 11:18

swjtu-zhanglei force-pushed the master-fix13 branch from e61b424 to a0479fbCompare September 11, 2025 11:18

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 11, 2025

dataroaring approved these changes Sep 12, 2025
View reviewed changes

dataroaring merged commit 2864924 into apache:masterSep 12, 2025
26 of 28 checks passed

github-actionsBot added the dev/3.1.x-conflict label Sep 12, 2025

swjtu-zhanglei mentioned this pull request Sep 22, 2025
branch-3.1: [feat](compaction) Make base compaction output rowset write filecache more adaptive #54694 #56278
Merged

morrySnow pushed a commit that referenced this pull request Sep 25, 2025
branch-3.1: [feat](compaction) Make base compaction output rowset wri…
57bfe70
…te filecache more adaptive #54694 (#56278) Cherry-picked from #54694

morrySnow added dev/3.1.2-merged and removed dev/3.1.x dev/3.1.x-conflict labels Sep 25, 2025

Uh oh!

Conversation

swjtu-zhanglei commented Aug 13, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

swjtu-zhanglei commented Aug 13, 2025

Uh oh!

Thearas commented Aug 13, 2025

Uh oh!

swjtu-zhanglei commented Aug 14, 2025

Uh oh!

doris-robot commented Aug 14, 2025

Uh oh!

doris-robot commented Aug 14, 2025

Uh oh!

doris-robot commented Aug 14, 2025

Uh oh!

hello-stephen commented Aug 14, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Uh oh!

swjtu-zhanglei commented Aug 29, 2025

Uh oh!

doris-robot commented Aug 29, 2025

Uh oh!

doris-robot commented Aug 29, 2025

Uh oh!

doris-robot commented Aug 29, 2025

Uh oh!

doris-robot commented Aug 29, 2025

BE UT Coverage Report

Uh oh!

swjtu-zhanglei commented Aug 29, 2025

Uh oh!

swjtu-zhanglei commented Aug 29, 2025

Uh oh!

doris-robot commented Aug 29, 2025

Uh oh!

doris-robot commented Aug 29, 2025

Uh oh!

doris-robot commented Aug 29, 2025

Uh oh!

doris-robot commented Sep 10, 2025

Uh oh!

doris-robot commented Sep 10, 2025

Uh oh!

doris-robot commented Sep 10, 2025

BE UT Coverage Report

Uh oh!

github-actionsBot commented Sep 10, 2025

Uh oh!

hello-stephen commented Sep 10, 2025

BE Regression && UT Coverage Report

Uh oh!

swjtu-zhanglei commented Sep 11, 2025

Uh oh!

doris-robot commented Sep 11, 2025

Uh oh!

doris-robot commented Sep 11, 2025

Uh oh!

doris-robot commented Sep 11, 2025

Uh oh!

hello-stephen commented Sep 11, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 11, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

swjtu-zhanglei commented Aug 13, 2025•
edited
Loading