[opt](routine-load) self-adaption backoff timeout by sollhui · Pull Request #32227 · apache/doris

sollhui · 2024-03-14T05:23:18Z

Proposed changes

Routine load is a long-term import task, assuming that the user has set a relatively small consumption interval, as the BE load changes, there may be a timeout for a certain period of time, leading to the problem of fake death.

Adaptive adjustment of timeout time to address the issue of possible changes in execution time caused by be load.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-14T05:23:22Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

sollhui · 2024-03-14T05:23:29Z

run buildall

doris-robot · 2024-03-14T05:43:00Z

TPC-H: Total hot run time: 38181 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a1ea74aa59dd63713cba5f1318f47ff8dfb0df3, data reload: false
------ Round 1 ----------------------------------
q1	17724	4233	4127	4127
q2	2021	159	144	144
q3	10670	1067	892	892
q4	7777	786	715	715
q5	7441	2612	2591	2591
q6	182	123	125	123
q7	1249	825	806	806
q8	9352	1986	1996	1986
q9	7061	6463	6420	6420
q10	8467	3468	3597	3468
q11	417	226	210	210
q12	611	311	293	293
q13	17801	2843	2839	2839
q14	266	248	251	248
q15	499	457	455	455
q16	482	394	389	389
q17	958	551	596	551
q18	7251	6525	6391	6391
q19	3724	1470	1391	1391
q20	563	292	289	289
q21	6285	3559	3623	3559
q22	355	335	294	294
Total cold run time: 111156 ms
Total hot run time: 38181 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4141	4061	4068	4061
q2	325	227	236	227
q3	2959	2846	2837	2837
q4	1788	1563	1529	1529
q5	5184	5219	5241	5219
q6	200	114	116	114
q7	2163	1823	1853	1823
q8	3156	3296	3310	3296
q9	8518	8544	8481	8481
q10	3666	3711	3696	3696
q11	525	438	438	438
q12	717	573	543	543
q13	16914	2866	2818	2818
q14	287	241	253	241
q15	489	446	454	446
q16	439	421	418	418
q17	1726	1464	1468	1464
q18	7511	7215	7065	7065
q19	1635	1542	1534	1534
q20	1896	1684	1708	1684
q21	4730	4607	4818	4607
q22	547	450	435	435
Total cold run time: 69516 ms
Total hot run time: 52976 ms

doris-robot · 2024-03-15T08:29:32Z

TPC-H: Total hot run time: 38503 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb67c1392629a3d6e308be0c264d09ed25f9f62c, data reload: false
------ Round 1 ----------------------------------
q1	17691	4566	4144	4144
q2	2027	158	159	158
q3	10688	1080	904	904
q4	7463	708	755	708
q5	7480	2709	2692	2692
q6	182	121	122	121
q7	1181	854	810	810
q8	9338	2003	1997	1997
q9	7150	6481	6395	6395
q10	8463	3525	3686	3525
q11	428	225	222	222
q12	631	306	296	296
q13	17822	2884	2842	2842
q14	276	252	255	252
q15	506	447	450	447
q16	511	391	381	381
q17	964	544	631	544
q18	7277	6577	6426	6426
q19	4488	1451	1498	1451
q20	560	292	288	288
q21	6233	3591	3584	3584
q22	360	318	316	316
Total cold run time: 111719 ms
Total hot run time: 38503 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4150	4072	4083	4072
q2	319	224	227	224
q3	2916	2872	2824	2824
q4	1848	1563	1567	1563
q5	5254	5277	5276	5276
q6	195	117	119	117
q7	2244	1833	1878	1833
q8	3156	3300	3307	3300
q9	8592	8583	8619	8583
q10	3738	3695	3693	3693
q11	558	451	439	439
q12	722	565	537	537
q13	16927	2894	2900	2894
q14	281	244	248	244
q15	487	455	455	455
q16	460	402	423	402
q17	1751	1494	1487	1487
q18	7584	7230	7242	7230
q19	3197	1530	1531	1530
q20	1922	1711	1722	1711
q21	4873	4605	4768	4605
q22	514	468	508	468
Total cold run time: 71688 ms
Total hot run time: 53487 ms

sollhui · 2024-03-15T12:42:09Z

run buildall

doris-robot · 2024-03-15T12:59:23Z

TPC-H: Total hot run time: 38415 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb67c1392629a3d6e308be0c264d09ed25f9f62c, data reload: false
------ Round 1 ----------------------------------
q1	17671	4486	4192	4192
q2	2031	167	146	146
q3	10650	1112	903	903
q4	7460	729	697	697
q5	7504	2664	2710	2664
q6	194	124	122	122
q7	1174	823	801	801
q8	9333	2032	1969	1969
q9	7114	6489	6427	6427
q10	8472	3492	3632	3492
q11	425	224	222	222
q12	633	310	301	301
q13	17816	2887	2862	2862
q14	282	262	249	249
q15	496	464	443	443
q16	479	391	382	382
q17	950	617	574	574
q18	7359	6593	6520	6520
q19	3059	1411	1449	1411
q20	548	291	276	276
q21	6273	3661	3468	3468
q22	355	318	294	294
Total cold run time: 110278 ms
Total hot run time: 38415 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4156	4116	4141	4116
q2	320	232	224	224
q3	2943	2802	2797	2797
q4	1815	1555	1530	1530
q5	5219	5269	5231	5231
q6	192	116	125	116
q7	2234	1842	1901	1842
q8	3183	3306	3292	3292
q9	8542	8560	8591	8560
q10	3710	3657	3654	3654
q11	541	462	436	436
q12	715	553	544	544
q13	16889	2876	2878	2876
q14	280	250	256	250
q15	494	441	451	441
q16	447	410	407	407
q17	1733	1473	1497	1473
q18	7717	7163	7144	7144
q19	1618	1571	1511	1511
q20	1911	1732	1714	1714
q21	4874	4810	4742	4742
q22	502	466	440	440
Total cold run time: 70035 ms
Total hot run time: 53340 ms

sollhui · 2024-03-18T09:47:31Z

run buildall

sollhui · 2024-03-19T08:27:19Z

run buildall

liaoxin01

LGTM

github-actions · 2024-03-20T07:11:19Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-20T07:11:21Z

PR approved by anyone and no changes requested.

* [opt](routine-load) self-adaption backoff timeout (#32227) * [fix](routine-load) fix timeout backoff can not work (#32661)

If IO/CPU resources are tight, routine load task is likely to timeout. The current method is self-adaption backoff #32227, but the problem is it will do some ineffective work to match proper timeout. For one routine load task, a better way to handle task is finishing executing instead of retry when resources are tight. Therefore, this pr increase timeout to make "task always finish even if it is slow when resources are tight".

) pick #40818 If IO/CPU resources are tight, routine load task is likely to timeout. The current method is self-adaption backoff #32227, but the problem is it will do some ineffective work to match proper timeout. For one routine load task, a better way to handle task is finishing executing instead of retry when resources are tight. Therefore, this pr increase timeout to make "task always finish even if it is slow when resources are tight".

) pick (#40818) If IO/CPU resources are tight, routine load task is likely to timeout. The current method is self-adaption backoff #32227, but the problem is it will do some ineffective work to match proper timeout. For one routine load task, a better way to handle task is finishing executing instead of retry when resources are tight. Therefore, this pr increase timeout to make "task always finish even if it is slow when resources are tight".

If IO/CPU resources are tight, routine load task is likely to timeout. The current method is self-adaption backoff apache#32227, but the problem is it will do some ineffective work to match proper timeout. For one routine load task, a better way to handle task is finishing executing instead of retry when resources are tight. Therefore, this pr increase timeout to make "task always finish even if it is slow when resources are tight".

self-adaption adjustment of timeout time
fb67c13

sollhuiand others added 2 commits March 18, 2024 16:07

Merge branch 'master' into timeout_opt
c855d4d

update
9157238

Merge branch 'master' into timeout_opt
05aa068

sollhui changed the title ~~[opt](routine-load) self-adaption adjustment of timeout time~~[opt](routine-load) self-adaption backoff timeoutMar 19, 2024

liaoxin01 approved these changes Mar 20, 2024
View reviewed changes

liaoxin01 added the dev/2.0.x label Mar 20, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 20, 2024

github-actionsBot added the reviewed label Mar 20, 2024

feifeifeimoon approved these changes Mar 20, 2024
View reviewed changes

liaoxin01 merged commit da09c3f into apache:masterMar 20, 2024

sollhui mentioned this pull request Mar 21, 2024
[branch-2.0](routine-load) self-adaption backoff timeout #32591
Merged

yiguolei added the 2.1.0-conflict label Mar 21, 2024

wm1581066 added dev/2.0.7-merged and removed dev/2.0.x labels Mar 21, 2024

sollhui mentioned this pull request Mar 22, 2024
[fix](routine-load) fix timeout backoff can not work #32661
Merged

yiguolei pushed a commit that referenced this pull request Mar 24, 2024
[branch-2.1](routine-load) self-adaption backoff timeout (#32734)
1b0cd4a
* [opt](routine-load) self-adaption backoff timeout (#32227) * [fix](routine-load) fix timeout backoff can not work (#32661)

yiguolei added dev/2.1.1-merged and removed 2.1.0-conflict labels Mar 25, 2024

sollhui mentioned this pull request Sep 13, 2024
[opt](routine load) optimize routine load timeout logic #40818
Merged

This was referenced Sep 23, 2024
[opt](routine load) optimize routine load timeout logic (#40818) #41135
Merged
[opt](routine load) optimize routine load timeout logic (#40818) #41137
Merged

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[opt](routine-load) self-adaption backoff timeout (apache#32227)
11fde1b

Navigation Menu

Uh oh!

[opt](routine-load) self-adaption backoff timeout - #32227

[opt](routine-load) self-adaption backoff timeout#32227
liaoxin01 merged 4 commits into
apache:masterfrom
sollhui:timeout_opt

sollhui commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 14, 2024

Uh oh!

sollhui commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

sollhui commented Mar 15, 2024

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

sollhui commented Mar 18, 2024

Uh oh!

sollhui commented Mar 19, 2024

Uh oh!

liaoxin01 left a comment

Uh oh!

github-actionsBot commented Mar 20, 2024

Uh oh!

github-actionsBot commented Mar 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

sollhui commented Mar 14, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 14, 2024

Uh oh!

sollhui commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

sollhui commented Mar 15, 2024

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

sollhui commented Mar 18, 2024

Uh oh!

sollhui commented Mar 19, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 20, 2024

Uh oh!

github-actionsBot commented Mar 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants