[opt](filecache) support config hot reload of file cache microbench in running state by HaolinGuan7211 · Pull Request #58922 · apache/doris

HaolinGuan7211 · 2025-12-10T16:20:00Z

No description provided.

Thearas · 2025-12-10T16:20:11Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

freemandealer · 2025-12-11T01:46:28Z

+
+ doris::Status release_bench_exec_env() {
+ auto* exec_env = doris::ExecEnv::GetInstance();
+ if (auto* file_cache_factory = exec_env->file_cache_factory()) {


add a condition ? if (... && config::clear_file_cache)

freemandealer · 2025-12-18T09:33:54Z

+
+ std::string custom_conffile = doris::config::custom_config_dir + "/be_custom.conf";
+ if (!doris::config::init(custom_conffile.c_str(), true, false, false)) {
+ return Status::InternalError("Error reading custom config file");


be_custom.conf is optional

freemandealer · 2025-12-18T09:44:22Z

@@ -835,10 +838,284 @@ struct Job {
 }
 };



microbenchService can be removed as we only use file_cache_action api

freemandealer

LGTM, greAAAAAt work!

freemandealer · 2025-12-18T12:03:15Z

run buildall

freemandealer · 2025-12-19T01:20:00Z

run compile

freemandealer · 2025-12-19T01:21:10Z

run beut

freemandealer · 2025-12-19T07:37:40Z

conflict is fixed in the new PR: #59186

github-actions · 2025-12-19T08:52:32Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-12-19T08:52:35Z

PR approved by anyone and no changes requested.

freemandealer · 2025-12-22T03:38:05Z

run buildall

freemandealer

LGTM

freemandealer

LGTM

doris-robot · 2025-12-22T05:37:16Z

TPC-H: Total hot run time: 35281 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d343035b484e624e0387b2ccf02da81045230ffc, data reload: false
------ Round 1 ----------------------------------
q1	17646	4214	4074	4074
q2	2069	367	245	245
q3	10130	1315	746	746
q4	10237	900	320	320
q5	8047	2318	1923	1923
q6	233	175	139	139
q7	1033	877	730	730
q8	9370	1450	1198	1198
q9	7361	5381	5347	5347
q10	6866	2394	1990	1990
q11	528	342	311	311
q12	699	719	586	586
q13	17794	3695	3063	3063
q14	291	297	275	275
q15	612	520	523	520
q16	696	685	647	647
q17	703	803	543	543
q18	8245	7192	6994	6994
q19	1114	962	615	615
q20	412	364	250	250
q21	4311	3938	3820	3820
q22	1049	984	945	945
Total cold run time: 109446 ms
Total hot run time: 35281 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4184	4119	4102	4102
q2	344	400	327	327
q3	2176	2672	2278	2278
q4	1360	1787	1325	1325
q5	4294	4934	4645	4645
q6	237	183	136	136
q7	2075	1912	1913	1912
q8	2861	2642	2582	2582
q9	7531	7555	7561	7555
q10	3059	3308	2810	2810
q11	645	555	601	555
q12	755	749	651	651
q13	3523	4313	3226	3226
q14	310	338	303	303
q15	573	525	533	525
q16	693	711	700	700
q17	1242	1418	1414	1414
q18	7791	7677	7564	7564
q19	909	867	920	867
q20	2014	2087	1823	1823
q21	4694	4249	4179	4179
q22	1103	1034	970	970
Total cold run time: 52373 ms
Total hot run time: 50449 ms

doris-robot · 2025-12-22T05:48:19Z

TPC-DS: Total hot run time: 179909 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d343035b484e624e0387b2ccf02da81045230ffc, data reload: false
query5	4639	594	451	451
query6	341	247	253	247
query7	4237	484	295	295
query8	321	248	239	239
query9	8778	2546	2532	2532
query10	523	380	331	331
query11	15648	14926	15146	14926
query12	178	126	120	120
query13	1270	489	404	404
query14	6050	3030	2784	2784
query14_1	2676	2654	2689	2654
query15	210	204	208	204
query16	844	440	439	439
query17	1101	680	581	581
query18	2434	430	329	329
query19	225	231	195	195
query20	115	116	113	113
query21	219	142	114	114
query22	4060	4063	3979	3979
query23	16656	16221	16088	16088
query23_1	16164	16163	16243	16163
query24	7337	1659	1215	1215
query24_1	1238	1221	1215	1215
query25	562	477	411	411
query26	1255	271	181	181
query27	2736	468	306	306
query28	4473	2126	2113	2113
query29	807	552	476	476
query30	317	244	214	214
query31	840	699	650	650
query32	75	73	74	73
query33	532	339	286	286
query34	887	899	539	539
query35	758	806	732	732
query36	890	943	819	819
query37	142	94	76	76
query38	2963	3059	2908	2908
query39	756	739	738	738
query39_1	700	701	703	701
query40	221	136	126	126
query41	64	62	63	62
query42	108	104	105	104
query43	424	436	400	400
query44	1305	753	740	740
query45	190	182	201	182
query46	857	982	607	607
query47	1681	1727	1652	1652
query48	325	326	257	257
query49	641	426	365	365
query50	654	309	220	220
query51	3840	3810	3800	3800
query52	109	119	98	98
query53	318	351	295	295
query54	309	252	242	242
query55	78	76	72	72
query56	304	329	305	305
query57	1171	1202	1094	1094
query58	263	246	252	246
query59	2434	2434	2406	2406
query60	321	308	286	286
query61	161	158	154	154
query62	776	698	683	683
query63	334	296	304	296
query64	4866	1294	1069	1069
query65	4018	3954	4001	3954
query66	1482	459	333	333
query67	15328	15199	14907	14907
query68	2893	1033	756	756
query69	444	365	312	312
query70	1036	1012	990	990
query71	310	302	285	285
query72	6388	5171	5180	5171
query73	516	595	301	301
query74	8985	9031	8763	8763
query75	3126	3203	2796	2796
query76	2890	1129	730	730
query77	355	386	311	311
query78	9428	9571	8724	8724
query79	1539	925	619	619
query80	1527	630	554	554
query81	542	269	236	236
query82	400	135	102	102
query83	361	260	232	232
query84	261	112	106	106
query85	931	496	451	451
query86	384	287	279	279
query87	3162	3193	3214	3193
query88	3216	2261	2267	2261
query89	470	431	388	388
query90	2042	174	152	152
query91	178	166	149	149
query92	74	66	68	66
query93	1123	910	564	564
query94	530	311	281	281
query95	564	383	302	302
query96	591	478	206	206
query97	2259	2304	2240	2240
query98	212	192	193	192
query99	1329	1337	1292	1292
Total cold run time: 254804 ms
Total hot run time: 179909 ms

doris-robot · 2025-12-22T05:53:24Z

ClickBench: Total hot run time: 27.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d343035b484e624e0387b2ccf02da81045230ffc, data reload: false
query1	0.05	0.05	0.05
query2	0.11	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.10
query5	0.27	0.26	0.27
query6	1.16	0.64	0.64
query7	0.04	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.52	0.50
query10	0.56	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.63	0.60	0.60
query14	1.00	0.97	0.98
query15	0.82	0.78	0.81
query16	0.39	0.39	0.41
query17	1.07	1.09	1.07
query18	0.23	0.22	0.21
query19	1.90	1.84	1.88
query20	0.02	0.02	0.01
query21	15.44	0.30	0.15
query22	4.69	0.05	0.05
query23	16.05	0.28	0.10
query24	0.96	0.45	1.28
query25	0.12	0.05	0.08
query26	0.15	0.14	0.13
query27	0.08	0.06	0.05
query28	5.56	1.22	1.02
query29	12.59	4.03	3.16
query30	0.28	0.15	0.12
query31	2.82	0.62	0.40
query32	3.24	0.55	0.45
query33	3.07	2.96	3.02
query34	16.88	5.16	4.57
query35	4.54	4.57	4.58
query36	0.67	0.50	0.49
query37	0.11	0.07	0.07
query38	0.08	0.05	0.04
query39	0.05	0.04	0.03
query40	0.16	0.14	0.13
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.74 s
Total hot run time: 27.46 s

doris-robot · 2025-12-22T07:09:19Z

BE UT Coverage Report

Increment line coverage 1.79% (1/56) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.41% (18923/35432)
Line Coverage	39.27% (175367/446557)
Region Coverage	33.82% (135664/401136)
Branch Coverage	34.73% (58475/168377)

hello-stephen · 2025-12-22T08:25:38Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/31) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.84% (25573/34634)
Line Coverage	61.24% (272682/445255)
Region Coverage	56.32% (228245/405296)
Branch Coverage	58.11% (98175/168939)

freemandealer · 2025-12-23T07:16:01Z

run external

freemandealer · 2025-12-23T07:16:12Z

run nonConcurrent

hello-stephen · 2025-12-23T09:08:20Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/31) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.23% (25017/34634)
Line Coverage	58.97% (262559/445255)
Region Coverage	53.90% (218451/405296)
Branch Coverage	55.43% (93647/168939)

freemandealer · 2025-12-24T06:58:36Z

run nonConcurrent

hello-stephen · 2025-12-24T09:23:42Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/31) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.23% (25016/34634)
Line Coverage	58.96% (262544/445255)
Region Coverage	53.88% (218390/405296)
Branch Coverage	55.42% (93632/168939)

hello-stephen · 2025-12-25T03:47:47Z

skip check_coverage

…in running state Pick apache#58922 to branch-4.1

…h config (pick#58922) (#62723) ### What problem does this PR solve? Issue Number: None Related PR: #58922 Problem Summary: Backport the file cache microbench config hot reload support to branch-4.0, including the file cache reload API and the supporting microbench service changes. ### Release note Support hot reload for the BE test file cache microbench configuration. ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [x] Other reason  Cherry-pick and conflict resolution only in this workspace. Local build/test was not run because `hooks/setup_worktree.sh` is unavailable here and `thirdparty/installed` is missing. - Behavior changed: - [ ] No. - [x] Yes. Adds hot reload support for the file cache microbench test tool. - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

… microbench in running state (#62595) ## Summary Pick #58922 to branch-4.1 ## Changes - Add `reload_file_cache` method to FileCacheFactory for hot reloading file cache configuration - Add `RELOAD` operation to FileCacheAction HTTP endpoint - Refactor file_cache_microbench to support config hot reload in running state - Introduce `BenchEnvManager` class to manage config loading and reload worker - Add `reload_config` and `show_reload_status` RPC endpoints - Improve read buffer management in JobManager ## Related Issue close#58922 Co-authored-by: LHG41278 <2020334843@qq.com>

HaolinGuan7211 marked this pull request as ready for review December 10, 2025 16:20

HaolinGuan7211 requested review from dataroaring and gavinchou as code owners December 10, 2025 16:20

freemandealer reviewed Dec 11, 2025
View reviewed changes

HaolinGuan7211 force-pushed the ghl/reload_fc_cfg branch from f41b8bc to 566a3ebCompare December 11, 2025 08:26

freemandealer approved these changes Dec 18, 2025
View reviewed changes

HaolinGuan7211 force-pushed the ghl/reload_fc_cfg branch from a978053 to 9bd9de7Compare December 18, 2025 11:43

freemandealer approved these changes Dec 18, 2025
View reviewed changes

gavinchou approved these changes Dec 18, 2025
View reviewed changes

gavinchou previously approved these changes Dec 18, 2025
View reviewed changes

resolve conflict
d343035

HaolinGuan7211 dismissed gavinchou’s stale review via d343035December 19, 2025 08:51

HaolinGuan7211 force-pushed the ghl/reload_fc_cfg branch from 9bd9de7 to d343035Compare December 19, 2025 08:51

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 19, 2025

github-actionsBot added the reviewed label Dec 19, 2025

HaolinGuan7211 requested a review from gavinchou December 19, 2025 17:02

freemandealer approved these changes Dec 22, 2025
View reviewed changes

gavinchou approved these changes Dec 25, 2025
View reviewed changes

gavinchou added dev/3.1.x dev/4.0.x labels Dec 25, 2025

gavinchou merged commit 8e12694 into apache:masterDec 25, 2025
32 of 33 checks passed

github-actionsBot added dev/3.1.x-conflict dev/4.0.x-conflict labels Dec 25, 2025

gavinchou pushed a commit to gavinchou/doris that referenced this pull request Apr 17, 2026
[opt](file-cache) support config hot reload of file cache microbench …
e3c830f
…in running state Pick apache#58922 to branch-4.1

gavinchou mentioned this pull request Apr 17, 2026
branch-4.1: [opt](file-cache) support config hot reload of file cache microbench in running state #62595
Merged

gavinchou added dev/4.1.x and removed dev/3.1.x dev/3.1.x-conflict dev/4.0.x dev/4.0.x-conflict labels Apr 20, 2026

github-actionsBot added the dev/4.1.x-conflict label Apr 20, 2026

gavinchou pushed a commit to gavinchou/doris that referenced this pull request Apr 22, 2026
[opt](file-cache) support config hot reload of file cache microbench …
2e49b76
…in running state Pick apache#58922 to branch-4.1

freemandealer mentioned this pull request Apr 22, 2026
branch-4.0: [feature](be) Support hot reload for file cache microbench config (pick#58922) #62723
Merged
16 tasks

yiguolei added dev/4.1.1-merged and removed dev/4.1.x dev/4.1.x-conflict labels Apr 25, 2026

yiguolei mentioned this pull request May 20, 2026
4.1.1 Release Notes #63426
Closed

morningman mentioned this pull request Jun 3, 2026
4.0.6 Release Notes #64065
Closed

Uh oh!

Conversation

HaolinGuan7211 commented Dec 10, 2025

Uh oh!

Thearas commented Dec 10, 2025

Uh oh!

freemandealerDec 11, 2025

Choose a reason for hiding this comment

Uh oh!

freemandealerDec 18, 2025

Choose a reason for hiding this comment

Uh oh!

freemandealerDec 18, 2025

Choose a reason for hiding this comment

Uh oh!

freemandealer left a comment

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Dec 18, 2025

Uh oh!

freemandealer commented Dec 19, 2025

Uh oh!

freemandealer commented Dec 19, 2025

Uh oh!

freemandealer commented Dec 19, 2025

Uh oh!

github-actionsBot commented Dec 19, 2025

Uh oh!

github-actionsBot commented Dec 19, 2025

Uh oh!

freemandealer commented Dec 22, 2025

Uh oh!

freemandealer left a comment

Choose a reason for hiding this comment

Uh oh!

freemandealer left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Dec 22, 2025

Uh oh!

doris-robot commented Dec 22, 2025

Uh oh!

doris-robot commented Dec 22, 2025

Uh oh!

doris-robot commented Dec 22, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 22, 2025

BE Regression && UT Coverage Report

Uh oh!

freemandealer commented Dec 23, 2025

Uh oh!

freemandealer commented Dec 23, 2025

Uh oh!

hello-stephen commented Dec 23, 2025

BE Regression && UT Coverage Report

Uh oh!

freemandealer commented Dec 24, 2025

Uh oh!

hello-stephen commented Dec 24, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Dec 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants