[fix](external catalog) Distinguish between functions used to refresh cache and change catalog properties by zy-kkk · Pull Request #56639 · apache/doris

zy-kkk · 2025-09-29T10:16:27Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

The resetToUninitialized method should only be called when altering catalog properties, as it resets the entire catalog to an uninitialized state. Previously, resetToUninitialized was mistakenly called in refreshMgr. Refresh should only refresh the cache, not the entire catalog. Resetting the entire catalog could cause certain clients to shut down during a cache refresh, leading to errors.

This PR mainly changes:

Change the onRefreshCache() method
This method will invalid db/table names cache and all meta cache
For refresh and hms listener, only call onRefreshCache()
For alter catalog, call resetToUninitialized()

TODO:
There is still a problem when we alter catalog, the ongoing task may be failed or stucked.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-09-29T10:16:36Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zy-kkk · 2025-09-29T12:55:58Z

run buildall

doris-robot · 2025-09-29T13:38:03Z

TPC-DS: Total hot run time: 190122 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8681721fdda615b687aff71a141880085583b2ba, data reload: false
query1	1062	434	407	407
query2	6566	1670	1663	1663
query3	6749	230	219	219
query4	26563	23519	23079	23079
query5	4626	657	487	487
query6	342	270	237	237
query7	4676	500	305	305
query8	340	295	263	263
query9	8724	2585	2556	2556
query10	520	353	292	292
query11	15255	15231	14832	14832
query12	188	125	117	117
query13	1732	590	461	461
query14	11771	9445	9520	9445
query15	230	185	184	184
query16	7776	747	575	575
query17	2107	810	652	652
query18	2181	446	381	381
query19	229	214	192	192
query20	147	128	146	128
query21	219	144	117	117
query22	4603	4592	4410	4410
query23	35038	33845	33650	33650
query24	8515	2500	2490	2490
query25	631	537	481	481
query26	1244	288	205	205
query27	3926	526	360	360
query28	4386	2308	2211	2211
query29	831	601	514	514
query30	299	220	199	199
query31	925	826	725	725
query32	80	73	69	69
query33	592	405	325	325
query34	806	853	526	526
query35	798	830	757	757
query36	979	1019	922	922
query37	124	107	86	86
query38	3511	3604	3412	3412
query39	1518	1382	1433	1382
query40	221	130	119	119
query41	60	63	58	58
query42	122	116	115	115
query43	501	505	482	482
query44	1380	842	830	830
query45	191	182	176	176
query46	862	991	644	644
query47	1748	1785	1703	1703
query48	408	442	319	319
query49	750	513	415	415
query50	674	707	403	403
query51	3856	3942	3968	3942
query52	107	111	98	98
query53	243	283	203	203
query54	601	594	533	533
query55	96	87	87	87
query56	352	351	310	310
query57	1196	1214	1113	1113
query58	295	290	286	286
query59	2558	2675	2603	2603
query60	352	384	325	325
query61	162	157	155	155
query62	809	728	667	667
query63	240	188	196	188
query64	4452	1161	836	836
query65	4023	3983	3937	3937
query66	1082	432	403	403
query67	15521	15353	15021	15021
query68	6981	959	609	609
query69	461	325	284	284
query70	1387	1344	1264	1264
query71	515	337	324	324
query72	6038	4869	4884	4869
query73	521	597	359	359
query74	8864	8860	8896	8860
query75	3674	3358	2881	2881
query76	3132	1166	811	811
query77	797	417	321	321
query78	9573	9740	8911	8911
query79	2015	843	595	595
query80	636	575	503	503
query81	503	263	235	235
query82	491	164	138	138
query83	269	276	253	253
query84	255	113	89	89
query85	909	563	515	515
query86	379	329	329	329
query87	3864	3759	3630	3630
query88	4036	2276	2269	2269
query89	417	335	296	296
query90	1961	227	221	221
query91	159	169	142	142
query92	89	71	66	66
query93	1725	987	655	655
query94	706	449	413	413
query95	409	327	319	319
query96	504	568	288	288
query97	2962	2968	2851	2851
query98	241	214	213	213
query99	1574	1450	1322	1322
Total cold run time: 279718 ms
Total hot run time: 190122 ms

doris-robot · 2025-09-29T13:43:32Z

ClickBench: Total hot run time: 29.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8681721fdda615b687aff71a141880085583b2ba, data reload: false
query1	0.06	0.04	0.05
query2	0.09	0.05	0.06
query3	0.26	0.08	0.09
query4	1.61	0.12	0.12
query5	0.27	0.27	0.26
query6	1.17	0.66	0.65
query7	0.04	0.03	0.02
query8	0.05	0.04	0.05
query9	0.64	0.52	0.52
query10	0.61	0.58	0.59
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.62	0.61
query14	1.02	1.03	1.03
query15	0.87	0.86	0.86
query16	0.41	0.40	0.41
query17	1.04	1.07	1.05
query18	0.22	0.20	0.20
query19	1.93	1.82	1.88
query20	0.01	0.02	0.02
query21	15.50	0.93	0.56
query22	0.75	1.13	0.69
query23	14.98	1.42	0.63
query24	7.54	1.44	0.51
query25	0.47	0.08	0.23
query26	0.71	0.17	0.13
query27	0.06	0.05	0.06
query28	8.55	1.34	0.92
query29	12.62	3.93	3.30
query30	0.28	0.14	0.11
query31	2.83	0.62	0.39
query32	3.26	0.55	0.47
query33	3.01	3.01	3.08
query34	16.27	5.47	4.81
query35	4.91	4.88	4.86
query36	0.69	0.52	0.49
query37	0.10	0.08	0.07
query38	0.06	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 104.21 s
Total hot run time: 29.97 s

hello-stephen · 2025-09-29T15:55:27Z

FE Regression Coverage Report

Increment line coverage 33.33% (3/9) 🎉
Increment coverage report
Complete coverage report

…talog properties

morningman · 2025-10-26T16:10:39Z

run buildall

doris-robot · 2025-10-26T17:12:56Z

ClickBench: Total hot run time: 28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f10156d10002c444aca3c330b315e79ae8fc54c9, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.06	0.06
query3	0.25	0.08	0.09
query4	1.61	0.11	0.12
query5	0.29	0.27	0.25
query6	1.16	0.66	0.63
query7	0.03	0.02	0.02
query8	0.05	0.04	0.05
query9	0.65	0.54	0.52
query10	0.60	0.58	0.58
query11	0.17	0.12	0.13
query12	0.15	0.13	0.16
query13	0.62	0.60	0.61
query14	1.02	1.02	1.01
query15	0.85	0.84	0.86
query16	0.41	0.41	0.39
query17	1.04	1.02	1.02
query18	0.22	0.21	0.20
query19	1.88	1.80	1.76
query20	0.02	0.01	0.02
query21	15.44	0.18	0.13
query22	5.16	0.07	0.06
query23	15.66	0.26	0.11
query24	2.31	1.17	0.78
query25	0.07	0.06	0.06
query26	0.14	0.14	0.13
query27	0.06	0.06	0.06
query28	5.28	1.18	0.94
query29	12.59	3.95	3.29
query30	0.30	0.15	0.11
query31	2.82	0.59	0.39
query32	3.25	0.55	0.47
query33	3.11	3.06	3.13
query34	15.95	5.15	4.55
query35	4.61	4.61	4.58
query36	0.68	0.51	0.49
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 99.15 s
Total hot run time: 28 s

hello-stephen · 2025-10-26T17:19:10Z

FE UT Coverage Report

Increment line coverage 82.61% (19/23) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-10-26T19:20:01Z

FE Regression Coverage Report

Increment line coverage 82.61% (19/23) 🎉
Increment coverage report
Complete coverage report

morningman · 2025-10-27T08:03:35Z

run buildall

doris-robot · 2025-10-27T08:40:46Z

ClickBench: Total hot run time: 29.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 599abbe49e50a1514d5cda53915d208576717e8b, data reload: false
query1	0.07	0.06	0.05
query2	0.10	0.06	0.06
query3	0.26	0.10	0.11
query4	1.62	0.13	0.13
query5	0.30	0.28	0.29
query6	1.19	0.68	0.69
query7	0.05	0.03	0.03
query8	0.07	0.05	0.05
query9	0.67	0.58	0.57
query10	0.64	0.64	0.64
query11	0.19	0.13	0.14
query12	0.19	0.14	0.15
query13	0.64	0.62	0.62
query14	1.07	1.03	1.03
query15	0.90	0.91	0.88
query16	0.42	0.43	0.45
query17	1.16	1.24	1.15
query18	0.24	0.22	0.22
query19	2.04	1.96	1.86
query20	0.02	0.02	0.01
query21	15.37	0.23	0.15
query22	4.89	0.08	0.06
query23	15.67	0.31	0.13
query24	2.49	0.65	0.62
query25	0.09	0.07	0.07
query26	0.18	0.17	0.17
query27	0.07	0.06	0.07
query28	4.56	1.23	0.95
query29	12.66	4.76	3.92
query30	0.30	0.16	0.13
query31	2.82	0.66	0.41
query32	3.24	0.58	0.49
query33	3.17	3.14	3.11
query34	15.89	5.23	4.67
query35	4.67	4.70	4.70
query36	0.69	0.54	0.52
query37	0.11	0.08	0.07
query38	0.08	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.10	0.04	0.04
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 99.22 s
Total hot run time: 29.62 s

hello-stephen · 2025-10-27T09:01:02Z

FE UT Coverage Report

Increment line coverage 82.61% (19/23) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-10-27T10:54:40Z

FE Regression Coverage Report

Increment line coverage 60.87% (14/23) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-10-28T03:41:30Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-10-28T03:41:32Z

PR approved by anyone and no changes requested.

… cache and change catalog properties (#56639) The `resetToUninitialized` method should only be called when altering catalog properties, as it resets the entire catalog to an uninitialized state. Previously, `resetToUninitialized` was mistakenly called in `refreshMgr`. Refresh should only refresh the cache, not the entire catalog. Resetting the entire catalog could cause certain clients to shut down during a cache refresh, leading to errors. This PR mainly changes: 1. Change the `onRefreshCache()` method This method will invalid db/table names cache and all meta cache 2. For `refresh` and `hms listener`, only call `onRefreshCache()` 3. For `alter catalog`, call `resetToUninitialized()` TODO: There is still a problem when we `alter catalog`, the ongoing task may be failed or stucked.

hello-stephen · 2025-10-28T04:01:40Z

FE Regression Coverage Report

Increment line coverage 60.87% (14/23) 🎉
Increment coverage report
Complete coverage report

…d to refresh cache and change catalog properties #56639 (#57400) Cherry-picked from #56639 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

…d to refresh cache and change catalog properties #56639 (#57399) Cherry-picked from #56639 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

… cache and change catalog properties (apache#56639) The `resetToUninitialized` method should only be called when altering catalog properties, as it resets the entire catalog to an uninitialized state. Previously, `resetToUninitialized` was mistakenly called in `refreshMgr`. Refresh should only refresh the cache, not the entire catalog. Resetting the entire catalog could cause certain clients to shut down during a cache refresh, leading to errors. This PR mainly changes: 1. Change the `onRefreshCache()` method This method will invalid db/table names cache and all meta cache 2. For `refresh` and `hms listener`, only call `onRefreshCache()` 3. For `alter catalog`, call `resetToUninitialized()` TODO: There is still a problem when we `alter catalog`, the ongoing task may be failed or stucked.

zy-kkk force-pushed the fix_refresh branch from 7dedd71 to 0251aebCompare September 29, 2025 12:45

zy-kkk added dev/3.1.x dev/4.0.x labels Sep 29, 2025

zy-kkk changed the title ~~[fix](external catalog) Remove refresh unnecessary reconstruction~~[fix](external catalog) Difference between refresh cache and alter catalog propertiesSep 29, 2025

zy-kkk changed the title ~~[fix](external catalog) Difference between refresh cache and alter catalog properties~~[fix](external catalog) Distinguish between functions used to refresh cache and change directory propertiesSep 29, 2025

zy-kkk changed the title ~~[fix](external catalog) Distinguish between functions used to refresh cache and change directory properties~~[fix](external catalog) Distinguish between functions used to refresh cache and change catalog propertiesSep 29, 2025

zy-kkk force-pushed the fix_refresh branch from 0251aeb to 8681721Compare September 29, 2025 12:55

zy-kkkand others added 3 commits October 26, 2025 23:59

[fix](external catalog) Difference between refresh cache and alter ca…
27cde48
…talog properties

1
18a3093

refactor
86a35e6

morningman force-pushed the fix_refresh branch from 8681721 to 86a35e6Compare October 26, 2025 16:00

morningman added 2 commits October 27, 2025 00:02

2
ac321c9

3
f10156d

morningman added 2 commits October 27, 2025 12:26

fix fe ut
7db3c6d

2
599abbe

morningman approved these changes Oct 28, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 28, 2025

github-actionsBot added the reviewed label Oct 28, 2025

CalvinKirs approved these changes Oct 28, 2025
View reviewed changes

zy-kkk merged commit 1baeaef into apache:masterOct 28, 2025
27 of 29 checks passed

zy-kkk deleted the fix_refresh branch October 28, 2025 03:51

github-actionsBot mentioned this pull request Oct 28, 2025
branch-3.1: [fix](external catalog) Distinguish between functions used to refresh cache and change catalog properties #56639 #57399
Merged

github-actionsBot mentioned this pull request Oct 28, 2025
branch-4.0: [fix](external catalog) Distinguish between functions used to refresh cache and change catalog properties #56639 #57400
Merged

yiguolei pushed a commit that referenced this pull request Oct 28, 2025
branch-4.0: [fix](external catalog) Distinguish between functions use…
6a1f031
…d to refresh cache and change catalog properties #56639 (#57400) Cherry-picked from #56639 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

yiguolei added dev/4.0.1-merged and removed dev/4.0.x labels Oct 28, 2025

morrySnow pushed a commit that referenced this pull request Oct 30, 2025
branch-3.1: [fix](external catalog) Distinguish between functions use…
4d20bc8
…d to refresh cache and change catalog properties #56639 (#57399) Cherry-picked from #56639 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

morrySnow added dev/3.1.3-merged and removed dev/3.1.x labels Oct 30, 2025

yiguolei mentioned this pull request Nov 5, 2025
4.0.1 Release Notes #57717
Closed

Uh oh!

Conversation

zy-kkk commented Sep 29, 2025• edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 29, 2025

Uh oh!

zy-kkk commented Sep 29, 2025

Uh oh!

doris-robot commented Sep 29, 2025

Uh oh!

doris-robot commented Sep 29, 2025

Uh oh!

hello-stephen commented Sep 29, 2025

FE Regression Coverage Report

Uh oh!

morningman commented Oct 26, 2025

Uh oh!

doris-robot commented Oct 26, 2025

Uh oh!

hello-stephen commented Oct 26, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 26, 2025

FE Regression Coverage Report

Uh oh!

morningman commented Oct 27, 2025

Uh oh!

doris-robot commented Oct 27, 2025

Uh oh!

hello-stephen commented Oct 27, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 27, 2025

FE Regression Coverage Report

Uh oh!

github-actionsBot commented Oct 28, 2025

Uh oh!

github-actionsBot commented Oct 28, 2025

Uh oh!

Uh oh!

hello-stephen commented Oct 28, 2025

FE Regression Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zy-kkk commented Sep 29, 2025•
edited by morningman
Loading