Skip to content

[fix](catalog) Fix lastUpdateTime not updated on refresh and add scheduled refresh logs - #58997

Merged
morningman merged 3 commits into
apache:masterfrom
zy-kkk:refresh_log
Dec 17, 2025
Merged

[fix](catalog) Fix lastUpdateTime not updated on refresh and add scheduled refresh logs#58997
morningman merged 3 commits into
apache:masterfrom
zy-kkk:refresh_log

Conversation

@zy-kkk

@zy-kkkzy-kkk commented Dec 12, 2025

Copy link
Copy Markdown
Member

Fix lastUpdateTime not being updated when catalog is refreshed (manual or scheduled).
Previously it was only set during initialization, now it reflects the actual last refresh time.

Also added info logs for scheduled catalog refresh to improve observability:

  • Log when catalog is added/removed from refresh map
  • Log when scheduled refresh is triggered and completed
  • Log catalog registration on FE startup

@zy-kkk

zy-kkk commented Dec 12, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@Thearas

Thearas commented Dec 12, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk

zy-kkk commented Dec 12, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

zhangstar333
zhangstar333 previously approved these changes Dec 12, 2025
View reviewed changes
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2025
@github-actions

github-actionsBot commented Dec 12, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Dec 12, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

morningman
morningman previously approved these changes Dec 12, 2025
View reviewed changes
@doris-robot

doris-robot commented Dec 12, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 35871 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7623f735103f7f31a3c6d41a5c619e5e95d3bf3, data reload: false
------ Round 1 ----------------------------------
q1	17670	4265	4037	4037
q2	2029	359	234	234
q3	10171	1314	728	728
q4	10203	803	312	312
q5	7525	2114	1930	1930
q6	181	170	137	137
q7	1012	845	710	710
q8	9361	1433	1155	1155
q9	7039	5353	5365	5353
q10	6852	2366	1973	1973
q11	531	323	284	284
q12	647	730	594	594
q13	17768	3684	3016	3016
q14	287	298	265	265
q15	587	519	518	518
q16	663	677	642	642
q17	680	831	502	502
q18	7334	7386	7950	7386
q19	1230	1049	635	635
q20	434	366	251	251
q21	4572	4254	4162	4162
q22	1118	1047	1071	1047
Total cold run time: 107894 ms
Total hot run time: 35871 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4296	4254	4320	4254
q2	325	404	342	342
q3	2448	2900	2473	2473
q4	1435	1875	1542	1542
q5	4766	4410	4288	4288
q6	214	176	132	132
q7	2010	1882	1836	1836
q8	2656	2468	2490	2468
q9	7552	7622	7476	7476
q10	3049	3123	2672	2672
q11	580	495	479	479
q12	655	726	526	526
q13	3245	3615	3041	3041
q14	259	282	252	252
q15	538	499	485	485
q16	610	670	599	599
q17	1137	1306	1376	1306
q18	7320	7108	7122	7108
q19	838	774	812	774
q20	1910	1974	1801	1801
q21	4583	4252	4224	4224
q22	1115	1010	962	962
Total cold run time: 51541 ms
Total hot run time: 49040 ms

@doris-robot

doris-robot commented Dec 12, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 179301 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d7623f735103f7f31a3c6d41a5c619e5e95d3bf3, data reload: false
query5	5109	602	491	491
query6	332	230	225	225
query7	4217	459	274	274
query8	313	270	245	245
query9	8780	2571	2563	2563
query10	563	376	335	335
query11	15659	14877	14754	14754
query12	185	131	115	115
query13	1271	534	411	411
query14	6361	3323	3008	3008
query14_1	2898	2945	2868	2868
query15	210	207	187	187
query16	901	483	455	455
query17	1140	722	631	631
query18	2720	449	367	367
query19	240	233	216	216
query20	130	116	113	113
query21	222	150	116	116
query22	3888	4037	3879	3879
query23	16455	16303	15911	15911
query23_1	16134	16078	16047	16047
query24	7344	1676	1273	1273
query24_1	1233	1244	1245	1244
query25	608	509	453	453
query26	1246	281	166	166
query27	2736	468	319	319
query28	4473	2149	2133	2133
query29	847	584	483	483
query30	322	246	219	219
query31	821	703	674	674
query32	79	84	82	82
query33	557	357	301	301
query34	907	894	553	553
query35	786	825	779	779
query36	874	927	858	858
query37	135	97	79	79
query38	2896	2852	2848	2848
query39	760	746	734	734
query39_1	690	691	693	691
query40	231	137	126	126
query41	78	65	65	65
query42	107	107	104	104
query43	421	439	423	423
query44	1378	738	737	737
query45	194	189	185	185
query46	883	982	605	605
query47	1631	1677	1600	1600
query48	317	330	248	248
query49	627	428	356	356
query50	669	289	228	228
query51	3808	4024	3771	3771
query52	108	115	100	100
query53	325	353	296	296
query54	298	267	262	262
query55	82	79	76	76
query56	305	303	310	303
query57	1149	1126	1071	1071
query58	311	265	261	261
query59	2421	2530	2415	2415
query60	308	322	295	295
query61	163	162	154	154
query62	719	687	641	641
query63	328	301	300	300
query64	4963	1363	1042	1042
query65	4016	3955	3978	3955
query66	1386	466	320	320
query67	15222	15043	14826	14826
query68	8330	1007	743	743
query69	515	357	318	318
query70	1133	1053	965	965
query71	364	320	297	297
query72	6079	4948	4886	4886
query73	660	563	315	315
query74	8886	8746	8582	8582
query75	3187	3134	2768	2768
query76	3997	1170	811	811
query77	540	410	326	326
query78	9445	9684	8906	8906
query79	1758	861	628	628
query80	730	654	578	578
query81	539	270	237	237
query82	207	135	108	108
query83	268	271	256	256
query84	256	126	98	98
query85	905	526	469	469
query86	387	324	285	285
query87	3219	3073	3080	3073
query88	3280	2292	2284	2284
query89	478	430	405	405
query90	2202	163	161	161
query91	180	176	149	149
query92	80	67	70	67
query93	1914	929	554	554
query94	481	311	297	297
query95	582	385	332	332
query96	597	474	212	212
query97	2252	2335	2221	2221
query98	216	204	193	193
query99	1321	1303	1245	1245
Total cold run time: 262825 ms
Total hot run time: 179301 ms

@doris-robot

doris-robot commented Dec 12, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 27.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d7623f735103f7f31a3c6d41a5c619e5e95d3bf3, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.04	0.04
query3	0.26	0.09	0.09
query4	1.60	0.10	0.10
query5	0.28	0.27	0.27
query6	1.17	0.66	0.63
query7	0.03	0.03	0.02
query8	0.04	0.04	0.05
query9	0.58	0.50	0.51
query10	0.56	0.55	0.57
query11	0.15	0.10	0.10
query12	0.16	0.12	0.12
query13	0.62	0.61	0.61
query14	0.98	0.99	0.98
query15	0.82	0.82	0.82
query16	0.39	0.40	0.39
query17	1.09	1.04	1.06
query18	0.24	0.22	0.22
query19	1.86	1.84	1.84
query20	0.01	0.01	0.01
query21	15.54	0.30	0.15
query22	4.87	0.05	0.06
query23	16.20	0.27	0.10
query24	0.94	0.68	0.18
query25	0.08	0.06	0.06
query26	0.15	0.13	0.14
query27	0.06	0.05	0.05
query28	3.75	1.22	1.03
query29	12.62	4.09	3.24
query30	0.29	0.14	0.12
query31	2.81	0.64	0.40
query32	3.24	0.55	0.46
query33	3.03	3.05	3.04
query34	16.99	5.23	4.58
query35	4.59	4.68	4.65
query36	0.66	0.51	0.50
query37	0.11	0.07	0.07
query38	0.07	0.05	0.04
query39	0.05	0.03	0.03
query40	0.17	0.16	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.05	0.04	0.04
Total cold run time: 97.4 s
Total hot run time: 27.52 s

@hello-stephen

hello-stephen commented Dec 12, 2025

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 17.39% (4/23) 🎉
Increment coverage report
Complete coverage report

@zy-kkk
zy-kkk dismissed stale reviews from morningman and zhangstar333 via adc0475December 16, 2025 02:58
@zy-kkk

zy-kkk commented Dec 16, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Dec 16, 2025
@doris-robot

doris-robot commented Dec 16, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 36290 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit adc0475067b888db96282dcc627398ca4ee58e69, data reload: false
------ Round 1 ----------------------------------
q1	17855	4283	4129	4129
q2	2191	352	235	235
q3	10292	1311	719	719
q4	10272	889	322	322
q5	7552	2158	1932	1932
q6	193	167	142	142
q7	1044	847	723	723
q8	9361	1449	1171	1171
q9	6992	5348	5292	5292
q10	7003	2391	1977	1977
q11	592	322	297	297
q12	705	722	586	586
q13	17770	3681	3067	3067
q14	283	292	284	284
q15	585	522	517	517
q16	663	676	627	627
q17	678	805	568	568
q18	7915	8042	7670	7670
q19	1555	978	626	626
q20	428	381	268	268
q21	4607	4265	4117	4117
q22	1133	1098	1021	1021
Total cold run time: 109669 ms
Total hot run time: 36290 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4335	4225	4282	4225
q2	327	379	343	343
q3	2415	2911	2593	2593
q4	1373	1806	1455	1455
q5	4610	4454	4505	4454
q6	230	176	130	130
q7	1983	1928	1767	1767
q8	2774	2506	2572	2506
q9	7638	7395	7388	7388
q10	3071	3259	2737	2737
q11	577	489	476	476
q12	631	723	566	566
q13	3311	3637	3046	3046
q14	274	275	260	260
q15	530	488	493	488
q16	621	643	611	611
q17	1093	1336	1369	1336
q18	7297	7087	6990	6990
q19	817	791	814	791
q20	1893	1968	1816	1816
q21	4703	4200	4130	4130
q22	1040	1081	993	993
Total cold run time: 51543 ms
Total hot run time: 49101 ms

@doris-robot

doris-robot commented Dec 16, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 178782 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit adc0475067b888db96282dcc627398ca4ee58e69, data reload: false
query5	4516	641	489	489
query6	339	214	208	208
query7	4210	486	280	280
query8	307	256	240	240
query9	8752	2576	2563	2563
query10	492	381	318	318
query11	15290	15292	14917	14917
query12	240	116	116	116
query13	1276	524	401	401
query14	6324	3275	3054	3054
query14_1	2879	2900	2891	2891
query15	213	198	187	187
query16	1036	477	463	463
query17	1187	720	602	602
query18	2528	450	353	353
query19	224	238	216	216
query20	126	119	113	113
query21	458	139	116	116
query22	4151	4167	3840	3840
query23	16493	16127	15976	15976
query23_1	16034	16157	15971	15971
query24	7340	1640	1230	1230
query24_1	1234	1215	1232	1215
query25	559	484	412	412
query26	1267	261	161	161
query27	2739	460	330	330
query28	4442	2152	2138	2138
query29	799	566	438	438
query30	352	242	212	212
query31	804	684	623	623
query32	113	74	68	68
query33	573	335	291	291
query34	901	906	539	539
query35	767	825	731	731
query36	873	910	818	818
query37	182	94	76	76
query38	2851	2854	2839	2839
query39	774	760	714	714
query39_1	720	697	697	697
query40	230	135	121	121
query41	67	66	61	61
query42	104	110	101	101
query43	418	435	419	419
query44	1332	749	764	749
query45	200	190	186	186
query46	868	983	618	618
query47	1660	1705	1605	1605
query48	341	323	255	255
query49	760	436	355	355
query50	662	288	225	225
query51	3806	3851	3830	3830
query52	107	113	97	97
query53	327	350	290	290
query54	305	259	244	244
query55	76	80	70	70
query56	303	299	296	296
query57	1137	1131	1069	1069
query58	282	250	249	249
query59	2403	2520	2318	2318
query60	326	310	290	290
query61	162	154	159	154
query62	716	677	621	621
query63	333	294	296	294
query64	5082	1349	998	998
query65	3997	3945	3941	3941
query66	1435	475	350	350
query67	15202	14987	14636	14636
query68	4741	1032	731	731
query69	500	351	310	310
query70	1055	975	975	975
query71	370	309	288	288
query72	6314	4989	5104	4989
query73	677	621	325	325
query74	8710	8713	8747	8713
query75	3170	3170	2772	2772
query76	3812	1153	768	768
query77	545	409	310	310
query78	9516	9654	8884	8884
query79	986	933	604	604
query80	1207	699	599	599
query81	550	279	237	237
query82	442	141	107	107
query83	402	265	254	254
query84	270	129	106	106
query85	1038	514	469	469
query86	407	317	281	281
query87	3025	3007	3018	3007
query88	3250	2298	2298	2298
query89	465	435	387	387
query90	1965	157	155	155
query91	172	169	147	147
query92	74	68	63	63
query93	1064	898	559	559
query94	529	325	286	286
query95	561	329	367	329
query96	589	460	212	212
query97	2270	2287	2216	2216
query98	201	197	190	190
query99	1307	1301	1221	1221
Total cold run time: 258606 ms
Total hot run time: 178782 ms

@doris-robot

doris-robot commented Dec 16, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 27.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit adc0475067b888db96282dcc627398ca4ee58e69, data reload: false
query1	0.05	0.04	0.04
query2	0.15	0.05	0.05
query3	0.28	0.09	0.08
query4	1.62	0.11	0.11
query5	0.28	0.25	0.26
query6	1.16	0.63	0.63
query7	0.04	0.03	0.02
query8	0.05	0.05	0.04
query9	0.60	0.51	0.51
query10	0.56	0.56	0.55
query11	0.17	0.10	0.12
query12	0.16	0.11	0.11
query13	0.63	0.60	0.61
query14	1.01	0.98	0.98
query15	0.82	0.79	0.80
query16	0.42	0.40	0.40
query17	1.00	1.01	1.00
query18	0.23	0.21	0.22
query19	1.90	1.72	1.77
query20	0.02	0.01	0.01
query21	15.46	0.27	0.15
query22	4.82	0.05	0.05
query23	16.16	0.30	0.11
query24	2.01	0.26	0.19
query25	0.06	0.10	0.05
query26	0.14	0.13	0.13
query27	0.10	0.05	0.05
query28	3.70	1.24	1.04
query29	12.55	4.07	3.23
query30	0.29	0.16	0.11
query31	2.81	0.62	0.38
query32	3.24	0.55	0.47
query33	2.90	3.02	3.04
query34	17.02	5.12	4.59
query35	4.57	4.58	4.66
query36	0.67	0.50	0.49
query37	0.13	0.06	0.07
query38	0.08	0.05	0.03
query39	0.07	0.04	0.04
query40	0.16	0.15	0.13
query41	0.10	0.03	0.03
query42	0.05	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 98.28 s
Total hot run time: 27.18 s

@hello-stephen

hello-stephen commented Dec 16, 2025

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 65.22% (15/23) 🎉
Increment coverage report
Complete coverage report

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 17, 2025
@github-actions

github-actionsBot commented Dec 17, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@morningman
morningman merged commit 345f473 into apache:masterDec 17, 2025
30 of 31 checks passed
github-actionsBot pushed a commit that referenced this pull request Dec 17, 2025
@zy-kkkYour Name
[fix](catalog) Fix lastUpdateTime not updated on refresh and add sche…
759e776
…duled refresh logs (#58997)
Fix `lastUpdateTime` not being updated when catalog is refreshed (manual
or scheduled).
Previously it was only set during initialization, now it reflects the
actual last refresh time.
Also added info logs for scheduled catalog refresh to improve
observability:
- Log when catalog is added/removed from refresh map
- Log when scheduled refresh is triggered and completed
- Log catalog registration on FE startup
@github-actionsgithub-actionsBot mentioned this pull request Dec 17, 2025
Merged
yiguolei pushed a commit that referenced this pull request Dec 18, 2025
@github-actions@zy-kkk
branch-4.0: [fix](catalog) Fix lastUpdateTime not updated on refresh …
88beb1a
…and add scheduled refresh logs #58997 (#59108)
Cherry-picked from #58997
Co-authored-by: zy-kkk <zhongyk10@gmail.com>
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Dec 18, 2025
@zy-kkk
[fix](catalog) Fix lastUpdateTime not updated on refresh and add sche…
550c4bd
…duled refresh logs (apache#58997)
Fix `lastUpdateTime` not being updated when catalog is refreshed (manual
or scheduled).
Previously it was only set during initialization, now it reflects the
actual last refresh time.
Also added info logs for scheduled catalog refresh to improve
observability:
- Log when catalog is added/removed from refresh map
- Log when scheduled refresh is triggered and completed
- Log catalog registration on FE startup
@zy-kkkzy-kkk mentioned this pull request Dec 18, 2025
Merged
morrySnow pushed a commit that referenced this pull request Dec 19, 2025
@zy-kkk
branch-3.1: [fix](catalog) Fix lastUpdateTime not updated on refresh …
5b22a1f
…and add scheduled refresh logs #58997 (#59143)
picked from #58997
@yiguoleiyiguolei mentioned this pull request Jan 23, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@hubgeterhubgeterhubgeter approved these changes
@zhangstar333zhangstar333zhangstar333 left review comments

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.xdev/3.1.4-mergeddev/4.0.3-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@zy-kkk@Thearas@doris-robot@hello-stephen@morningman@hubgeter@zhangstar333@yiguolei@morrySnow