Skip to content

[opt](routine-load) optimize allocate task to be algorithm for load balance - #32021

Merged
liaoxin01 merged 1 commit into
apache:masterfrom
sollhui:allocate_task
Mar 20, 2024
Merged

[opt](routine-load) optimize allocate task to be algorithm for load balance#32021
liaoxin01 merged 1 commit into
apache:masterfrom
sollhui:allocate_task

Conversation

@sollhui

@sollhuisollhui commented Mar 9, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

The task always running on one BE may cause the BE to run too many tasks simultaneously while the other BE is not running few task. Here, a simple limitation is added to optimize this situation.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented Mar 9, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@sollhui

sollhui commented Mar 9, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Mar 9, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@sollhui

sollhui commented Mar 15, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Mar 15, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38261 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72a0e9d98388da6dff99073a7bd49e63918bafaf, data reload: false
------ Round 1 ----------------------------------
q1	17624	4935	4136	4136
q2	2026	151	141	141
q3	10605	1089	916	916
q4	6741	757	703	703
q5	7467	2613	2619	2613
q6	183	123	126	123
q7	1253	821	800	800
q8	9321	1979	2012	1979
q9	7129	6420	6422	6420
q10	8510	3490	3616	3490
q11	423	226	222	222
q12	709	295	305	295
q13	17788	2865	2867	2865
q14	277	243	254	243
q15	499	454	451	451
q16	517	399	397	397
q17	948	563	554	554
q18	7190	6518	6346	6346
q19	4921	1417	1385	1385
q20	532	292	284	284
q21	6497	3613	3581	3581
q22	361	342	317	317
Total cold run time: 111521 ms
Total hot run time: 38261 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4171	4114	4114	4114
q2	321	231	222	222
q3	2996	2829	2845	2829
q4	1874	1550	1602	1550
q5	5250	5263	5240	5240
q6	195	117	121	117
q7	2221	1832	1817	1817
q8	3152	3312	3308	3308
q9	8573	8526	8551	8526
q10	3710	3679	3656	3656
q11	534	458	448	448
q12	715	547	555	547
q13	16922	2881	2857	2857
q14	276	268	249	249
q15	481	460	445	445
q16	444	401	431	401
q17	1751	1505	1461	1461
q18	7670	7176	7018	7018
q19	1637	1514	1568	1514
q20	1938	1714	1721	1714
q21	4843	4567	4794	4567
q22	530	462	460	460
Total cold run time: 70204 ms
Total hot run time: 53060 ms

liaoxin01
liaoxin01 approved these changes Mar 16, 2024
View reviewed changes

@liaoxin01liaoxin01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 16, 2024
@github-actions

github-actionsBot commented Mar 16, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@liaoxin01

liaoxin01 commented Mar 18, 2024

Copy link
Copy Markdown
Contributor

run p0

@liaoxin01

liaoxin01 commented Mar 18, 2024

Copy link
Copy Markdown
Contributor

run performance

@liaoxin01
liaoxin01 merged commit e7d2b38 into apache:masterMar 20, 2024
yiguolei pushed a commit that referenced this pull request Mar 21, 2024
@sollhui@Doris-Extras
[opt](routine-load) optimize allocate task to be algorithm for load b…
acde3ae
…alance (#32021)
yiguolei pushed a commit that referenced this pull request Mar 21, 2024
@sollhui@Doris-Extras
[opt](routine-load) optimize allocate task to be algorithm for load b…
22782fe
…alance (#32021)
@yiguoleiyiguolei mentioned this pull request Mar 24, 2024
Closed
@sollhuisollhui mentioned this pull request Jul 8, 2024
Merged
liaoxin01 pushed a commit that referenced this pull request Jul 8, 2024
@sollhui
[opt](routine-load) optimize allocate task to be algorithm for load b…
598c46d
…alance (#32021) (#37434)
@xiaokangxiaokang mentioned this pull request Jul 14, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@sollhui@Baymine
[opt](routine-load) optimize allocate task to be algorithm for load b…
078d78e
…alance (apache#32021) (apache#37434)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@sollhui@HappenLee
[opt](routine-load) optimize allocate task to be algorithm for load b…
62207a5
…alance (apache#32021)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01liaoxin01liaoxin01 approved these changes
+2 more reviewers
@kaijchenkaijchenkaijchen approved these changes
@feifeifeimoonfeifeifeimoonfeifeifeimoon approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.13-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sollhui@doris-robot@liaoxin01@kaijchen@feifeifeimoon