[Fix](cloud) Cloud enable fe deploy mode from master-observers to multi-followers by deardeng · Pull Request #45255 · apache/doris

deardeng · 2024-12-10T10:21:36Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

IN CLOUD

add some cloud metaservice api regression test
prohibit one node add to multi cluster repeatedly
supports switching from master observer mode to multi follower mode, and compatible with upgrading from existing master-observers deployment mode to multi-follwers mode
when adding an FE node(add fe cluster, add fe node), there is a 5-minute protection time and the FE node cannot be droped(drop fe cluster, drop fe node)
drop the node. If the node cannot be found, an error will occur and the code will return 404 instead of the previous silent processing
If the MS node starts with 127.0.0.1 IP and the prohibit_use_loopback_addresses is set to true(default), the MS node will not be able to start, ms log will find enable check prohibit use loopback addr, but localhost=127.0.0.1, so exit(-1)

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…ti-follers

doris-robot · 2024-12-10T10:21:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions · 2024-12-10T10:26:32Z

clang-tidy review says "All clean, LGTM! 👍"

deardeng · 2024-12-10T10:46:17Z

run buildall

deardeng · 2024-12-10T11:30:42Z

run buildall

github-actions · 2024-12-10T11:33:56Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-10T11:50:19Z

TPC-H: Total hot run time: 39925 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be63162b4fc91141f8f9ef1ced2c639c17c8b2fe, data reload: false
------ Round 1 ----------------------------------
q1	17573	7377	7373	7373
q2	2048	188	169	169
q3	10568	1095	1209	1095
q4	10552	717	730	717
q5	7601	2715	2715	2715
q6	234	148	152	148
q7	998	647	594	594
q8	9248	1836	1925	1836
q9	6617	6443	6559	6443
q10	6965	2309	2284	2284
q11	465	263	248	248
q12	431	231	223	223
q13	17784	2991	2963	2963
q14	241	204	215	204
q15	580	532	538	532
q16	663	603	581	581
q17	980	533	585	533
q18	7161	6782	6630	6630
q19	1341	1072	1013	1013
q20	467	180	189	180
q21	3991	3266	3124	3124
q22	381	323	320	320
Total cold run time: 106889 ms
Total hot run time: 39925 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7298	7238	7191	7191
q2	329	228	242	228
q3	2881	2741	2921	2741
q4	2095	1828	1792	1792
q5	5677	5650	5633	5633
q6	227	141	145	141
q7	2218	1770	1800	1770
q8	3340	3572	3494	3494
q9	8955	9122	9040	9040
q10	3597	3548	3527	3527
q11	619	517	522	517
q12	817	600	603	600
q13	11435	3285	3222	3222
q14	319	271	288	271
q15	578	521	537	521
q16	686	657	653	653
q17	1829	1628	1632	1628
q18	8278	7761	7554	7554
q19	1691	1609	1604	1604
q20	2089	1898	1834	1834
q21	5594	5565	5540	5540
q22	648	577	633	577
Total cold run time: 71200 ms
Total hot run time: 60078 ms

deardeng · 2024-12-10T11:59:17Z

run cloudut

doris-robot · 2024-12-10T12:02:55Z

TPC-DS: Total hot run time: 197298 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be63162b4fc91141f8f9ef1ced2c639c17c8b2fe, data reload: false
query1	1255	946	933	933
query2	6212	2050	2034	2034
query3	11021	4591	4399	4399
query4	66485	28631	23660	23660
query5	4889	486	458	458
query6	412	183	178	178
query7	5479	302	294	294
query8	322	236	242	236
query9	8368	2703	2690	2690
query10	435	244	240	240
query11	16933	15265	15933	15265
query12	155	101	101	101
query13	1455	422	407	407
query14	10307	7178	7734	7178
query15	207	192	192	192
query16	7177	475	508	475
query17	1099	595	588	588
query18	1909	317	311	311
query19	204	167	165	165
query20	130	108	113	108
query21	213	106	112	106
query22	4646	4432	4382	4382
query23	35027	34301	34612	34301
query24	5389	2426	2513	2426
query25	484	387	414	387
query26	648	148	156	148
query27	1881	280	289	280
query28	4434	2429	2444	2429
query29	702	439	409	409
query30	221	159	165	159
query31	1028	849	841	841
query32	64	53	57	53
query33	441	289	299	289
query34	930	508	551	508
query35	871	748	772	748
query36	1079	963	959	959
query37	127	73	79	73
query38	4582	4414	4357	4357
query39	1514	1459	1477	1459
query40	229	103	99	99
query41	44	43	44	43
query42	117	103	96	96
query43	538	513	491	491
query44	1202	841	840	840
query45	195	171	175	171
query46	1190	731	728	728
query47	2006	1951	1950	1950
query48	405	324	310	310
query49	724	404	409	404
query50	845	395	393	393
query51	7344	7253	7278	7253
query52	102	86	84	84
query53	267	182	178	178
query54	532	412	394	394
query55	79	77	76	76
query56	247	233	233	233
query57	1269	1095	1107	1095
query58	212	209	208	208
query59	3189	2930	2953	2930
query60	263	237	235	235
query61	114	133	102	102
query62	781	671	697	671
query63	215	189	184	184
query64	1353	650	660	650
query65	3281	3205	3178	3178
query66	696	308	297	297
query67	15927	15659	15876	15659
query68	3935	573	562	562
query69	451	254	253	253
query70	1215	1149	1176	1149
query71	362	253	250	250
query72	6366	4200	4148	4148
query73	747	360	355	355
query74	10323	8963	9092	8963
query75	3382	2678	2672	2672
query76	1883	1037	1125	1037
query77	492	286	274	274
query78	10343	9443	9451	9443
query79	1380	594	591	591
query80	943	445	434	434
query81	484	251	226	226
query82	1305	122	118	118
query83	248	140	140	140
query84	287	71	70	70
query85	875	295	307	295
query86	348	294	270	270
query87	4708	4585	4650	4585
query88	3318	2201	2164	2164
query89	422	304	294	294
query90	2014	187	186	186
query91	139	104	101	101
query92	64	48	52	48
query93	1770	526	532	526
query94	809	299	285	285
query95	349	262	254	254
query96	608	278	278	278
query97	2831	2644	2661	2644
query98	215	193	195	193
query99	1589	1312	1325	1312
Total cold run time: 316306 ms
Total hot run time: 197298 ms

doris-robot · 2024-12-10T12:08:39Z

ClickBench: Total hot run time: 33.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be63162b4fc91141f8f9ef1ced2c639c17c8b2fe, data reload: false
query1	0.04	0.03	0.02
query2	0.09	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.08	0.08
query5	0.42	0.40	0.44
query6	1.17	0.67	0.67
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.57	0.51	0.50
query10	0.56	0.56	0.57
query11	0.16	0.12	0.12
query12	0.17	0.14	0.13
query13	0.62	0.60	0.60
query14	2.71	2.78	2.71
query15	0.93	0.85	0.85
query16	0.38	0.36	0.39
query17	1.01	1.03	1.01
query18	0.20	0.19	0.19
query19	2.01	1.90	2.06
query20	0.01	0.02	0.01
query21	15.35	0.67	0.66
query22	4.79	6.92	1.90
query23	18.30	1.46	1.38
query24	2.11	0.24	0.23
query25	0.16	0.09	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.18	1.16	1.15
query29	12.63	3.45	3.41
query30	0.24	0.07	0.06
query31	2.87	0.41	0.39
query32	3.22	0.50	0.49
query33	3.01	3.04	3.20
query34	16.74	4.50	4.47
query35	4.56	4.53	4.54
query36	0.67	0.49	0.49
query37	0.21	0.18	0.16
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.97 s
Total hot run time: 33.39 s

deardeng · 2024-12-13T04:11:44Z

run buildall

github-actions · 2024-12-13T04:15:06Z

clang-tidy review says "All clean, LGTM! 👍"

deardeng · 2024-12-16T12:12:12Z

run buildall

github-actions · 2024-12-16T12:16:41Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-23T03:53:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-23T03:53:04Z

PR approved by anyone and no changes requested.

gavinchou · 2025-01-01T07:22:10Z

run buildall

github-actions · 2025-01-01T07:22:45Z

PR approved by at least one committer and no changes requested.

doris-robot · 2025-01-01T08:20:03Z

TPC-H: Total hot run time: 32560 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b62b015448a3958fefd9f2cae68c004f145966ee, data reload: false
------ Round 1 ----------------------------------
q1	17844	6189	6096	6096
q2	2049	327	167	167
q3	10664	1222	746	746
q4	10266	865	438	438
q5	8569	2211	1983	1983
q6	208	178	148	148
q7	891	758	607	607
q8	9243	1376	1179	1179
q9	5276	4888	4908	4888
q10	6784	2317	1889	1889
q11	498	280	248	248
q12	342	360	224	224
q13	17751	3609	2954	2954
q14	242	232	219	219
q15	565	508	486	486
q16	635	603	588	588
q17	587	858	330	330
q18	7074	6481	6312	6312
q19	2789	970	568	568
q20	306	316	185	185
q21	2835	2210	1994	1994
q22	376	339	311	311
Total cold run time: 105794 ms
Total hot run time: 32560 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6349	6255	6213	6213
q2	236	328	238	238
q3	2299	2684	2313	2313
q4	1398	1879	1373	1373
q5	4365	4749	4926	4749
q6	186	178	143	143
q7	2055	2008	1811	1811
q8	2698	2779	2677	2677
q9	7417	7340	7384	7340
q10	3058	3364	2747	2747
q11	578	502	494	494
q12	656	782	676	676
q13	3358	3841	3125	3125
q14	290	312	298	298
q15	567	514	516	514
q16	667	663	653	653
q17	1258	1719	1248	1248
q18	7604	7477	7440	7440
q19	886	1193	1098	1098
q20	2033	2030	1961	1961
q21	5655	5232	4813	4813
q22	617	638	597	597
Total cold run time: 54230 ms
Total hot run time: 52521 ms

doris-robot · 2025-01-01T08:32:30Z

TPC-DS: Total hot run time: 197888 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b62b015448a3958fefd9f2cae68c004f145966ee, data reload: false
query1	1320	979	927	927
query2	6478	2395	2378	2378
query3	11108	4867	4819	4819
query4	32736	26624	23715	23715
query5	4356	607	436	436
query6	284	227	187	187
query7	3985	493	310	310
query8	308	288	239	239
query9	9595	2726	2726	2726
query10	501	312	245	245
query11	17924	15354	15222	15222
query12	153	114	104	104
query13	1585	520	409	409
query14	9828	8046	7498	7498
query15	263	206	192	192
query16	8763	614	469	469
query17	1639	773	601	601
query18	2151	437	325	325
query19	244	178	156	156
query20	119	117	117	117
query21	200	137	108	108
query22	4597	4615	4388	4388
query23	34911	33766	33831	33766
query24	6683	2300	2300	2300
query25	479	459	395	395
query26	760	282	153	153
query27	2029	478	337	337
query28	5851	2508	2449	2449
query29	634	555	419	419
query30	225	218	150	150
query31	982	964	876	876
query32	82	60	60	60
query33	486	347	299	299
query34	761	884	544	544
query35	795	825	745	745
query36	1049	1071	976	976
query37	121	100	78	78
query38	4234	4375	4251	4251
query39	1501	1436	1526	1436
query40	205	115	101	101
query41	46	43	44	43
query42	124	108	107	107
query43	530	537	496	496
query44	1331	827	825	825
query45	189	172	170	170
query46	886	1096	671	671
query47	2047	2049	1975	1975
query48	431	402	316	316
query49	711	480	416	416
query50	638	666	389	389
query51	7362	7125	7155	7125
query52	104	102	96	96
query53	236	252	187	187
query54	480	496	411	411
query55	81	76	83	76
query56	256	264	256	256
query57	1238	1245	1164	1164
query58	238	234	220	220
query59	3209	3354	3142	3142
query60	264	270	238	238
query61	109	125	104	104
query62	874	810	774	774
query63	228	211	193	193
query64	3219	1039	660	660
query65	3354	3227	3280	3227
query66	799	419	325	325
query67	16466	16095	15507	15507
query68	9375	754	517	517
query69	480	291	252	252
query70	1205	1182	1174	1174
query71	441	300	247	247
query72	6342	3872	3843	3843
query73	665	773	365	365
query74	10277	9172	8973	8973
query75	4646	3171	2649	2649
query76	4920	1203	761	761
query77	941	366	278	278
query78	10051	10107	9439	9439
query79	3239	898	663	663
query80	744	517	426	426
query81	480	265	230	230
query82	656	151	123	123
query83	204	159	144	144
query84	289	96	66	66
query85	781	364	310	310
query86	343	325	307	307
query87	4565	4662	4556	4556
query88	4249	2243	2216	2216
query89	464	339	288	288
query90	1835	184	182	182
query91	131	134	113	113
query92	67	52	54	52
query93	1356	869	526	526
query94	654	415	285	285
query95	321	260	244	244
query96	488	606	282	282
query97	2826	2835	2705	2705
query98	228	201	192	192
query99	1696	1565	1461	1461
Total cold run time: 300334 ms
Total hot run time: 197888 ms

doris-robot · 2025-01-01T08:37:50Z

ClickBench: Total hot run time: 31.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b62b015448a3958fefd9f2cae68c004f145966ee, data reload: false
query1	0.03	0.04	0.03
query2	0.09	0.05	0.04
query3	0.23	0.05	0.06
query4	1.64	0.08	0.08
query5	0.42	0.39	0.41
query6	1.17	0.66	0.67
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.55	0.49	0.50
query10	0.54	0.57	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.61	0.59	0.60
query14	2.71	2.88	2.78
query15	0.92	0.84	0.85
query16	0.38	0.38	0.38
query17	1.04	1.08	1.07
query18	0.19	0.19	0.18
query19	1.84	1.75	2.04
query20	0.01	0.02	0.01
query21	15.35	0.96	0.66
query22	0.78	0.80	0.70
query23	14.94	1.51	0.67
query24	2.21	0.36	0.24
query25	0.15	0.08	0.08
query26	0.28	0.19	0.18
query27	0.09	0.08	0.08
query28	13.42	1.75	1.15
query29	12.65	4.06	3.38
query30	0.25	0.08	0.06
query31	2.82	0.62	0.39
query32	3.22	0.57	0.48
query33	3.13	3.26	3.15
query34	16.82	5.23	4.60
query35	4.61	4.61	4.63
query36	0.62	0.48	0.49
query37	0.20	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.05
query40	0.18	0.13	0.13
query41	0.09	0.05	0.06
query42	0.07	0.05	0.05
query43	0.06	0.05	0.04
Total cold run time: 104.91 s
Total hot run time: 31.71 s

Hastyshell

LGTM

…ti-followers (apache#45255) IN CLOUD 1. add some cloud metaservice api regression test 2. prohibit one node add to multi cluster repeatedly 3. supports switching from master observer mode to multi follower mode, and compatible with upgrading from existing master-observers deployment mode to multi-follwers mode 4. when adding an FE node(add fe cluster, add fe node), there is a 5-minute protection time and the FE node cannot be droped(drop fe cluster, drop fe node) 5. drop the node. If the node cannot be found, an error will occur and the code will return 404 instead of the previous silent processing 6. If the MS node starts with 127.0.0.1 IP and the `prohibit_use_loopback_addresses` is set to true(default), the MS node will not be able to start, ms log will find `enable check prohibit use loopback addr, but localhost=127.0.0.1, so exit(-1)`

…rvers to multi-followers #45255 (#47215) cherry pick from #45255

### What problem does this PR solve? Related PR: #45255

) ### What problem does this PR solve? Related PR: apache#45255

[Fix](cloud) Cloud enable fe deploy mode from master-observers to mul…
8a8e0df
…ti-follers

fix ut
be63162

dataroaring added the dev/3.0.x label Dec 11, 2024

gavinchou reviewed Dec 11, 2024
View reviewed changes

Comment threadcloud/src/common/config.h Outdated

gavinchou reviewed Dec 11, 2024
View reviewed changes

Comment threadcloud/src/common/network_util.cpp

fix review
d782bd9

add
93ac740

gavinchou previously approved these changes Dec 23, 2024
View reviewed changes

gavinchou reviewed Dec 23, 2024
View reviewed changes

Comment threadcloud/src/common/config.h Outdated

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024

github-actionsBot added the reviewed label Dec 23, 2024

gavinchou reviewed Dec 23, 2024
View reviewed changes

Comment threadcloud/src/meta-service/injection_point_http.cpp Outdated

gavinchou reviewed Dec 23, 2024
View reviewed changes

Comment threadgensrc/proto/cloud.proto

gavinchou reviewed Dec 23, 2024
View reviewed changes

Comment threadcloud/src/common/config.h Outdated

fix
920dbac

Apply suggestions from code review
abb2038

gavinchou dismissed their stale review via abb2038January 1, 2025 06:50

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jan 1, 2025

gavinchou reviewed Jan 1, 2025
View reviewed changes

Comment threadcloud/test/fdb_injection_test.cpp Outdated

Update cloud/test/fdb_injection_test.cpp
b62b015

gavinchou approved these changes Jan 1, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 1, 2025

Hastyshell approved these changes Jan 3, 2025
View reviewed changes

gavinchou merged commit 3939c99 into apache:masterJan 3, 2025

github-actionsBot added the dev/3.0.x-conflict label Jan 3, 2025

gavinchou mentioned this pull request Jan 18, 2025
[fix](meta-service) Read recycle rowset key before removing when commit rowset #46657
Closed
16 tasks

deardeng mentioned this pull request Jan 20, 2025
branch-3.0: [Fix](cloud) Cloud enable fe deploy mode from master-observers to multi-followers #45255 #47215
Merged

gavinchou pushed a commit that referenced this pull request Jan 23, 2025
branch-3.0: [Fix](cloud) Cloud enable fe deploy mode from master-obse…
c4c526a
…rvers to multi-followers #45255 (#47215) cherry pick from #45255

wm1581066 added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jan 23, 2025

gavinchou mentioned this pull request Feb 7, 2025
[fix](UT) Fix cloud UT network_util_test and resource_test #47582
Merged
16 tasks

hello-stephen pushed a commit that referenced this pull request Feb 8, 2025
[fix](UT) Fix cloud UT network_util_test and resource_test (#47582)
c689789
### What problem does this PR solve? Related PR: #45255

github-actionsBot pushed a commit that referenced this pull request Feb 8, 2025
[fix](UT) Fix cloud UT network_util_test and resource_test (#47582)
ea255db
### What problem does this PR solve? Related PR: #45255

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
[fix](UT) Fix cloud UT network_util_test and resource_test (apache#47582
18fce35
) ### What problem does this PR solve? Related PR: apache#45255

xy720 mentioned this pull request Apr 29, 2025
[enhancement](cloud) support param to drop fe cluster not in safe time #50535
Closed
16 tasks

xy720 mentioned this pull request May 7, 2025
[enhancement](cloud) support param to drop fe cluster not in safe time #50680
Closed
16 tasks

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
[fix](UT) Fix cloud UT network_util_test and resource_test (apache#47582
629a14e
) ### What problem does this PR solve? Related PR: apache#45255

Uh oh!

Conversation

deardeng commented Dec 10, 2024• edited by gavinchou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Dec 10, 2024

Uh oh!

github-actionsBot commented Dec 10, 2024

Uh oh!

deardeng commented Dec 10, 2024

Uh oh!

deardeng commented Dec 10, 2024

Uh oh!

github-actionsBot commented Dec 10, 2024

Uh oh!

doris-robot commented Dec 10, 2024

Uh oh!

deardeng commented Dec 10, 2024

Uh oh!

doris-robot commented Dec 10, 2024

Uh oh!

doris-robot commented Dec 10, 2024

Uh oh!

Uh oh!

Uh oh!

deardeng commented Dec 13, 2024

Uh oh!

github-actionsBot commented Dec 13, 2024

Uh oh!

deardeng commented Dec 16, 2024

Uh oh!

github-actionsBot commented Dec 16, 2024

Uh oh!

Uh oh!

github-actionsBot commented Dec 23, 2024

Uh oh!

github-actionsBot commented Dec 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gavinchou commented Jan 1, 2025

Uh oh!

github-actionsBot commented Jan 1, 2025

Uh oh!

doris-robot commented Jan 1, 2025

Uh oh!

doris-robot commented Jan 1, 2025

Uh oh!

doris-robot commented Jan 1, 2025

Uh oh!

Hastyshell left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

deardeng commented Dec 10, 2024•
edited by gavinchou
Loading