[fix](catalog) Clean up refresh thread when deleting catalog by zy-kkk · Pull Request #57680 · apache/doris

zy-kkk · 2025-11-04T07:29:03Z

since #49621
Since ExternalCatalog no longer calls methods in the interface when onClose, we need to manually clean up the values in refreshmanager when deleting the catalog.

Thearas · 2025-11-04T07:29:09Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zy-kkk · 2025-11-04T07:29:33Z

3.1 Unified fix in this PR #57671

zy-kkk · 2025-11-04T07:29:39Z

run buildall

doris-robot · 2025-11-04T08:12:26Z

TPC-DS: Total hot run time: 189821 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0974b5a0bdac1a769a9285dbca6fefb7e844c88, data reload: false
query1	1075	400	393	393
query2	6581	1697	1692	1692
query3	6757	224	215	215
query4	26016	23245	23116	23116
query5	4733	627	472	472
query6	343	257	239	239
query7	4648	492	296	296
query8	315	279	259	259
query9	8748	2577	2578	2577
query10	509	342	290	290
query11	15549	15208	14932	14932
query12	193	120	114	114
query13	1692	580	446	446
query14	12582	9271	9474	9271
query15	213	189	184	184
query16	7746	703	526	526
query17	1653	788	641	641
query18	2098	541	378	378
query19	260	234	198	198
query20	149	143	169	143
query21	214	137	127	127
query22	4758	4875	4938	4875
query23	35136	34010	33837	33837
query24	8251	2560	2551	2551
query25	601	549	460	460
query26	1270	298	164	164
query27	2874	510	365	365
query28	4432	2252	2204	2204
query29	851	674	505	505
query30	300	225	199	199
query31	954	862	759	759
query32	80	75	72	72
query33	583	515	384	384
query34	1580	850	531	531
query35	827	840	826	826
query36	1059	1025	945	945
query37	132	107	84	84
query38	3512	3555	3523	3523
query39	1458	1444	1407	1407
query40	214	127	114	114
query41	62	60	58	58
query42	123	109	111	109
query43	481	493	466	466
query44	1218	739	731	731
query45	183	175	184	175
query46	874	1000	633	633
query47	1774	1779	1690	1690
query48	396	418	314	314
query49	771	518	421	421
query50	660	688	412	412
query51	3867	3933	3834	3834
query52	108	105	102	102
query53	230	259	188	188
query54	309	303	273	273
query55	87	85	82	82
query56	324	314	310	310
query57	1186	1190	1121	1121
query58	285	282	276	276
query59	2578	2719	2514	2514
query60	348	337	337	337
query61	160	159	153	153
query62	813	726	701	701
query63	233	197	192	192
query64	4410	1153	877	877
query65	4004	3943	3964	3943
query66	1090	438	355	355
query67	15339	15078	15027	15027
query68	7114	954	595	595
query69	460	316	285	285
query70	1324	1271	1259	1259
query71	493	331	324	324
query72	5861	5006	4908	4908
query73	513	560	358	358
query74	8877	9321	8621	8621
query75	3650	3380	2880	2880
query76	3232	1158	794	794
query77	786	411	322	322
query78	9509	9732	8920	8920
query79	2230	843	585	585
query80	650	557	513	513
query81	519	254	228	228
query82	464	156	126	126
query83	265	263	249	249
query84	260	117	90	90
query85	944	495	440	440
query86	392	307	302	302
query87	3741	3718	3579	3579
query88	4121	2221	2198	2198
query89	384	337	298	298
query90	1932	218	221	218
query91	164	164	141	141
query92	80	69	62	62
query93	2143	985	625	625
query94	673	438	344	344
query95	410	329	323	323
query96	497	573	276	276
query97	2927	2958	2888	2888
query98	249	222	207	207
query99	1370	1387	1255	1255
Total cold run time: 279093 ms
Total hot run time: 189821 ms

doris-robot · 2025-11-04T08:17:31Z

ClickBench: Total hot run time: 27.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0974b5a0bdac1a769a9285dbca6fefb7e844c88, data reload: false
query1	0.06	0.05	0.05
query2	0.08	0.05	0.05
query3	0.25	0.08	0.08
query4	1.60	0.12	0.11
query5	0.26	0.26	0.24
query6	1.19	0.64	0.64
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.61	0.51	0.50
query10	0.57	0.56	0.57
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.60
query14	1.02	0.99	1.00
query15	0.85	0.84	0.84
query16	0.40	0.38	0.38
query17	1.02	1.04	1.02
query18	0.22	0.20	0.20
query19	1.89	1.86	1.74
query20	0.02	0.01	0.02
query21	15.44	0.19	0.13
query22	4.99	0.06	0.05
query23	15.68	0.26	0.09
query24	1.84	1.10	0.64
query25	0.07	0.05	0.06
query26	0.14	0.13	0.14
query27	0.07	0.06	0.05
query28	4.28	1.16	0.93
query29	12.60	3.85	3.24
query30	0.27	0.14	0.11
query31	2.81	0.60	0.39
query32	3.24	0.55	0.47
query33	3.11	3.02	3.05
query34	15.85	5.19	4.56
query35	4.55	4.53	4.59
query36	0.68	0.50	0.49
query37	0.10	0.06	0.06
query38	0.06	0.04	0.03
query39	0.04	0.02	0.02
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.19 s
Total hot run time: 27.5 s

zy-kkk · 2025-11-04T08:40:45Z

run buildall

doris-robot · 2025-11-04T09:20:59Z

TPC-DS: Total hot run time: 189312 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95fe8a5bdb48f369b6baf9d41fe3c15d475825c4, data reload: false
query1	1057	443	398	398
query2	6573	1694	1724	1694
query3	6763	231	220	220
query4	26254	23391	23117	23117
query5	4395	624	458	458
query6	330	250	245	245
query7	4642	491	295	295
query8	302	266	257	257
query9	8718	2602	2592	2592
query10	512	348	291	291
query11	15399	15051	14787	14787
query12	186	120	123	120
query13	1712	573	440	440
query14	11272	9289	9339	9289
query15	206	194	174	174
query16	7699	680	509	509
query17	1519	794	636	636
query18	2046	441	327	327
query19	222	209	179	179
query20	137	145	143	143
query21	230	149	128	128
query22	4700	4576	4568	4568
query23	34439	34072	33560	33560
query24	9005	2537	2525	2525
query25	661	531	466	466
query26	1333	330	174	174
query27	3950	516	364	364
query28	4325	2274	2224	2224
query29	833	728	499	499
query30	298	235	203	203
query31	971	856	767	767
query32	86	72	70	70
query33	616	405	343	343
query34	836	885	520	520
query35	832	847	784	784
query36	956	1019	958	958
query37	127	111	86	86
query38	3621	3536	3496	3496
query39	1455	1402	1398	1398
query40	215	123	113	113
query41	60	55	58	55
query42	119	108	109	108
query43	480	494	468	468
query44	1219	727	727	727
query45	183	176	168	168
query46	880	986	633	633
query47	1759	1813	1736	1736
query48	390	424	320	320
query49	769	497	444	444
query50	639	689	397	397
query51	3840	3897	3805	3805
query52	112	105	102	102
query53	241	259	191	191
query54	297	286	292	286
query55	92	88	81	81
query56	314	313	304	304
query57	1173	1202	1117	1117
query58	285	278	275	275
query59	2489	2722	2552	2552
query60	331	342	331	331
query61	156	158	158	158
query62	785	732	673	673
query63	224	197	192	192
query64	4432	1168	866	866
query65	4060	3977	3932	3932
query66	1108	430	337	337
query67	15468	15114	14941	14941
query68	8541	937	597	597
query69	482	314	288	288
query70	1373	1260	1303	1260
query71	507	345	318	318
query72	6025	4959	4974	4959
query73	691	600	361	361
query74	8781	9110	9017	9017
query75	4085	3367	2788	2788
query76	3673	1182	743	743
query77	795	390	305	305
query78	9490	9794	8943	8943
query79	2292	833	615	615
query80	649	561	515	515
query81	520	265	228	228
query82	464	154	135	135
query83	281	263	249	249
query84	252	111	95	95
query85	915	498	445	445
query86	391	311	310	310
query87	3746	3763	3652	3652
query88	3831	2237	2236	2236
query89	384	345	304	304
query90	1894	218	220	218
query91	163	170	138	138
query92	87	68	61	61
query93	1892	997	647	647
query94	686	435	314	314
query95	404	327	302	302
query96	484	577	282	282
query97	2919	2986	2869	2869
query98	237	210	212	210
query99	1453	1403	1292	1292
Total cold run time: 279725 ms
Total hot run time: 189312 ms

doris-robot · 2025-11-04T09:26:05Z

ClickBench: Total hot run time: 27.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95fe8a5bdb48f369b6baf9d41fe3c15d475825c4, data reload: false
query1	0.06	0.05	0.04
query2	0.09	0.04	0.04
query3	0.26	0.08	0.09
query4	1.60	0.12	0.12
query5	0.28	0.27	0.25
query6	1.14	0.65	0.63
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.60	0.53	0.51
query10	0.59	0.58	0.57
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.60	0.61
query14	1.01	0.99	1.00
query15	0.85	0.82	0.83
query16	0.40	0.41	0.42
query17	1.07	1.06	0.99
query18	0.22	0.19	0.20
query19	1.89	1.86	1.81
query20	0.01	0.02	0.02
query21	15.46	0.19	0.13
query22	5.01	0.07	0.05
query23	15.68	0.28	0.11
query24	2.72	1.10	0.77
query25	0.08	0.06	0.05
query26	0.14	0.14	0.13
query27	0.06	0.05	0.05
query28	4.89	1.14	0.92
query29	12.58	3.83	3.24
query30	0.28	0.13	0.12
query31	2.82	0.58	0.37
query32	3.23	0.54	0.48
query33	3.03	3.01	3.05
query34	15.93	5.16	4.52
query35	4.57	4.51	4.60
query36	0.68	0.50	0.50
query37	0.09	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.04
query40	0.17	0.15	0.13
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.8 s
Total hot run time: 27.69 s

hello-stephen · 2025-11-04T11:08:30Z

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-11-05T09:35:04Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-05T09:35:06Z

PR approved by anyone and no changes requested.

since #49621 Since ExternalCatalog no longer calls methods in the interface when onClose, we need to manually clean up the values in refreshmanager when deleting the catalog.

hello-stephen · 2025-11-05T09:56:09Z

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

…log #57680 (#57734) Cherry-picked from #57680 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

…57680) since apache#49621 Since ExternalCatalog no longer calls methods in the interface when onClose, we need to manually clean up the values in refreshmanager when deleting the catalog.

[fix](catalog) Clean up refresh thread when deleting catalog
f0974b5

zy-kkk added dev/3.1.x dev/4.0.x labels Nov 4, 2025

add test
95fe8a5

morningman approved these changes Nov 5, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 5, 2025

github-actionsBot added the reviewed label Nov 5, 2025

hubgeter approved these changes Nov 5, 2025
View reviewed changes

morningman merged commit 7a77c08 into apache:masterNov 5, 2025
26 of 28 checks passed

github-actionsBot mentioned this pull request Nov 5, 2025
branch-3.1: [fix](catalog) Clean up refresh thread when deleting catalog #57680 #57733
Closed

github-actionsBot mentioned this pull request Nov 5, 2025
branch-4.0: [fix](catalog) Clean up refresh thread when deleting catalog #57680 #57734
Merged

morningman mentioned this pull request Nov 5, 2025
branch-3.1: [fix](catalog) Prevent removing catalog from refresh queue on CREATE/ALTER #57671
Merged

zy-kkk deleted the fix_remove_refresh branch November 6, 2025 03:35

zy-kkk added dev/3.1.3-merged and removed dev/3.1.x labels Nov 6, 2025

yiguolei pushed a commit that referenced this pull request Nov 8, 2025
branch-4.0: [fix](catalog) Clean up refresh thread when deleting cata…
b4dbae8
…log #57680 (#57734) Cherry-picked from #57680 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

yiguolei added dev/4.0.2-merged and removed dev/4.0.x labels Nov 8, 2025

Navigation Menu

Uh oh!

[fix](catalog) Clean up refresh thread when deleting catalog - #57680

[fix](catalog) Clean up refresh thread when deleting catalog#57680
morningman merged 2 commits into
apache:masterfrom
zy-kkk:fix_remove_refresh

zy-kkk commented Nov 4, 2025•
edited
Loading

Uh oh!

Thearas commented Nov 4, 2025

Uh oh!

zy-kkk commented Nov 4, 2025

Uh oh!

zy-kkk commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

zy-kkk commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

hello-stephen commented Nov 4, 2025

Uh oh!

github-actionsBot commented Nov 5, 2025

Uh oh!

github-actionsBot commented Nov 5, 2025

Uh oh!

Uh oh!

hello-stephen commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

zy-kkk commented Nov 4, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thearas commented Nov 4, 2025

Uh oh!

zy-kkk commented Nov 4, 2025

Uh oh!

zy-kkk commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

zy-kkk commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

doris-robot commented Nov 4, 2025

Uh oh!

hello-stephen commented Nov 4, 2025

FE Regression Coverage Report

Uh oh!

github-actionsBot commented Nov 5, 2025

Uh oh!

github-actionsBot commented Nov 5, 2025

Uh oh!

Uh oh!

hello-stephen commented Nov 5, 2025

FE Regression Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zy-kkk commented Nov 4, 2025•
edited
Loading