[fix](fe) Retry meta service too busy responses

[mymeiyi]

When a meta service RPC returned successfully but carried MS_TOO_BUSY in the response status, FE returned that response immediately and let callers handle a transient overload as a normal business result.
This change passes a typed status extractor into executeRequest, retries MS_TOO_BUSY responses with the existing retry counter and backoff, and throws RpcException if all attempts still return MS_TOO_BUSY.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[mymeiyi]

/review

[Copilot]

Pull request overview

This PR improves FE’s cloud MetaService RPC resilience by treating MS_TOO_BUSY as a transient overload signal (even when the RPC itself succeeds), retrying with the existing retry/backoff loop, and surfacing an RpcException only after retries are exhausted.

Changes:

• Extend MetaServiceClientWrapper.executeRequest / executeWithMetrics to accept a typed statusExtractor and retry responses whose status code is MS_TOO_BUSY.
• Update all MetaServiceProxy RPC wrappers to pass the correct getStatus extractor.
• Add FE unit tests validating: no shutdown on MS_TOO_BUSY, retry-to-success, and failure after retries.

Review Checkpoint Conclusions (from skill checklist)

• Goal & completeness: The change matches the PR description (retry MS_TOO_BUSY responses) and adds targeted unit tests demonstrating the new behavior.
• Scope: Focused to MetaServiceProxy retry behavior and tests; no unrelated refactors observed.
• Concurrency/lifecycle: No new concurrency primitives introduced; retry loop behavior remains synchronous and uses existing Thread.sleep backoff.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
fe/fe-core/src/main/java/org/apache/doris/cloud/rpc/MetaServiceProxy.java	Adds status-based retry for MS_TOO_BUSY by passing a response-status extractor through the existing retry/metrics wrapper.
fe/fe-core/src/test/java/org/apache/doris/cloud/rpc/MetaServiceProxyTest.java	Updates existing tests for new API signature and adds coverage for MS_TOO_BUSY retry/failure behavior.

Comments suppressed due to low confidence (1)

fe/fe-core/src/main/java/org/apache/doris/cloud/rpc/MetaServiceProxy.java:249

• Log message contains a typo: "meta servive" → "meta service".

 requestFailed = true;
LOG.warn("failed to request meta servive trycnt {}", tried, e);
if (tried >= maxRetries) {

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Review complete. I found one correctness gap: the new MS_TOO_BUSY response retry is wired through the shared synchronous wrapper, but MetaServiceProxy.getInstance() still bypasses that wrapper and can return a status-bearing too-busy response directly. I left one inline comment.

Critical checkpoint conclusions:

• Goal/test: wrapper-based synchronous RPCs now retry MS_TOO_BUSY and have focused unit coverage, but getInstance is a parallel status-bearing RPC path without the new retry behavior or a test.
• Scope: the change is mostly focused and does not add config, storage format, or protocol fields.
• Concurrency/lifecycle: no new shared mutable state or lock ordering change found; the retry path intentionally avoids marking MS_TOO_BUSY as a client failure.
• Compatibility: mixed FE/meta-service compatibility is not materially changed because MS_TOO_BUSY already exists in the proto.
• Parallel paths: getInstance is the uncovered parallel path and is the blocking issue.
• Testing/style: static checks found all new status extractors refer to response types with a status field and no changed Java lines over 120 columns. I did not run FE tests because this checkout is missing .worktree_initialized, thirdparty/installed, and thirdparty/installed/bin/protoc.

User focus: no additional user-provided focus points were present.

Subagent conclusions: optimizer-rewrite proposed OPT-1, which became MAIN-1 and is submitted inline. tests-session-config proposed TSC-1 as a duplicate of OPT-1 with test coverage evidence, merged into MAIN-1. Convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same ledger/comment set.

[mymeiyi]

run buildall

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 6.80% (40/588) 🎉
Increment coverage report
Complete coverage report

[github-actions]

PR approved by at least one committer and no changes requested.

[mymeiyi]

run external

[mymeiyi]

run performance

[hello-stephen]

TPC-H: Total hot run time: 29542 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b22162226799f04097af73439f138ccc95a3cec5, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17763	4094	4003	4003
q2	2020	322	203	203
q3	10277	1429	850	850
q4	4724	465	341	341
q5	7835	857	573	573
q6	245	170	135	135
q7	795	818	621	621
q8	10384	1559	1608	1559
q9	5778	4425	4412	4412
q10	6796	1802	1546	1546
q11	508	340	313	313
q12	694	545	422	422
q13	18081	3824	2732	2732
q14	266	262	253	253
q15	q16	791	777	718	718
q17	928	1052	912	912
q18	6921	5648	5725	5648
q19	1194	1236	1047	1047
q20	780	646	569	569
q21	5705	2690	2394	2394
q22	428	351	291	291
Total cold run time: 102913 ms
Total hot run time: 29542 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4348	4279	4284	4279
q2	284	322	208	208
q3	4572	4964	4400	4400
q4	2062	2152	1358	1358
q5	4468	4365	4373	4365
q6	228	172	126	126
q7	2147	1820	1634	1634
q8	2513	2176	2093	2093
q9	8010	7873	7805	7805
q10	4740	4754	4273	4273
q11	560	591	371	371
q12	739	766	549	549
q13	3223	3625	3033	3033
q14	322	317	283	283
q15	q16	739	746	654	654
q17	1353	1318	1338	1318
q18	8129	7449	7112	7112
q19	1089	1052	1060	1052
q20	2253	2210	1954	1954
q21	5268	4572	4403	4403
q22	516	458	396	396
Total cold run time: 57563 ms
Total hot run time: 51666 ms

[hello-stephen]

TPC-DS: Total hot run time: 182134 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b22162226799f04097af73439f138ccc95a3cec5, data reload: false
query5	4310	655	501	501
query6	460	226	200	200
query7	4869	644	355	355
query8	340	188	172	172
query9	8750	4125	4127	4125
query10	491	348	303	303
query11	5943	2416	2168	2168
query12	176	104	103	103
query13	1291	598	469	469
query14	6334	5359	5020	5020
query14_1	4383	4325	4458	4325
query15	226	213	182	182
query16	1043	476	406	406
query17	1141	758	622	622
query18	2740	480	354	354
query19	214	202	155	155
query20	115	107	107	107
query21	234	162	138	138
query22	13698	13538	13414	13414
query23	17406	16528	16146	16146
query23_1	16459	16296	16367	16296
query24	7724	1795	1321	1321
query24_1	1356	1324	1320	1320
query25	592	493	404	404
query26	1333	356	220	220
query27	2671	636	392	392
query28	4410	2068	2057	2057
query29	1106	641	520	520
query30	342	267	235	235
query31	1117	1115	1004	1004
query32	103	67	63	63
query33	550	338	269	269
query34	1168	1174	664	664
query35	768	799	684	684
query36	1422	1420	1245	1245
query37	156	113	96	96
query38	1874	1718	1658	1658
query39	938	925	893	893
query39_1	920	876	884	876
query40	251	209	143	143
query41	67	61	62	61
query42	92	95	93	93
query43	335	337	284	284
query44	1422	798	782	782
query45	206	188	180	180
query46	1064	1227	766	766
query47	2412	2351	2270	2270
query48	406	415	302	302
query49	581	430	310	310
query50	1090	428	320	320
query51	10870	10601	10533	10533
query52	85	87	77	77
query53	266	296	208	208
query54	280	253	222	222
query55	76	70	72	70
query56	298	289	291	289
query57	1446	1412	1320	1320
query58	285	291	255	255
query59	1594	1679	1477	1477
query60	298	271	259	259
query61	154	145	151	145
query62	703	655	593	593
query63	251	206	208	206
query64	2785	1063	888	888
query65	4915	4795	4782	4782
query66	1772	521	394	394
query67	29921	29591	29443	29443
query68	3243	1543	897	897
query69	399	304	272	272
query70	1077	1005	924	924
query71	349	314	302	302
query72	3120	2728	2393	2393
query73	860	782	425	425
query74	5117	4934	4788	4788
query75	2621	2602	2230	2230
query76	2309	1200	808	808
query77	381	390	310	310
query78	12298	12311	11742	11742
query79	1436	1243	778	778
query80	1313	557	477	477
query81	550	327	277	277
query82	628	168	131	131
query83	393	329	300	300
query84	272	165	135	135
query85	981	600	513	513
query86	420	293	265	265
query87	1830	1821	1759	1759
query88	3760	2851	2810	2810
query89	457	416	354	354
query90	1918	205	206	205
query91	201	190	176	176
query92	67	64	57	57
query93	1667	1535	934	934
query94	724	366	305	305
query95	778	581	469	469
query96	1041	766	370	370
query97	2699	2716	2561	2561
query98	224	207	207	207
query99	1143	1182	1027	1027
Total cold run time: 267776 ms
Total hot run time: 182134 ms

[hello-stephen]

ClickBench: Total hot run time: 24.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b22162226799f04097af73439f138ccc95a3cec5, data reload: false
query1	0.01	0.01	0.01
query2	0.09	0.04	0.05
query3	0.25	0.14	0.14
query4	1.62	0.15	0.14
query5	0.24	0.25	0.22
query6	1.26	1.06	1.07
query7	0.04	0.00	0.01
query8	0.05	0.04	0.03
query9	0.38	0.31	0.32
query10	0.57	0.55	0.57
query11	0.19	0.14	0.13
query12	0.18	0.15	0.14
query13	0.48	0.48	0.47
query14	1.02	1.02	1.01
query15	0.63	0.60	0.60
query16	0.32	0.33	0.30
query17	1.10	1.10	1.10
query18	0.23	0.20	0.21
query19	1.99	1.88	1.95
query20	0.02	0.01	0.01
query21	15.42	0.21	0.13
query22	4.96	0.05	0.05
query23	16.15	0.30	0.13
query24	2.93	0.44	0.32
query25	0.10	0.04	0.04
query26	0.71	0.20	0.15
query27	0.05	0.04	0.04
query28	3.52	0.94	0.56
query29	12.51	4.12	3.21
query30	0.28	0.16	0.15
query31	2.78	0.61	0.31
query32	3.22	0.60	0.48
query33	3.15	3.20	3.21
query34	15.59	4.33	3.54
query35	3.58	3.47	3.50
query36	0.55	0.43	0.43
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.17	0.17
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.69 s
Total hot run time: 24.95 s

dismiss