[improve](routine load) delay schedule EOF tasks to avoid too many small transactions by sollhui · Pull Request #39975 · apache/doris

sollhui · 2024-08-27T07:14:31Z

We encountered a scenario where a large number of small transactions were generated, resulting in an impact on query performance:
Kafka's data comes in batches of very small data every very short time, which leads to tasks being frequently scheduled and ending very quickly, resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive EOF, which would not delay data consumption, for perceiving EOF indicates that the consumption speed is greater than the production speed.

doris-robot · 2024-08-27T07:14:35Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-08-27T07:17:01Z

run buildall

sollhui · 2024-08-27T07:35:18Z

run buildall

doris-robot · 2024-08-27T08:36:30Z

TPC-H: Total hot run time: 38466 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4378dbf5bc9a276126dc45a34421b713bff0c17, data reload: false
------ Round 1 ----------------------------------
q1	17751	4522	4441	4441
q2	2631	183	178	178
q3	11118	1203	1196	1196
q4	10490	732	740	732
q5	7755	2923	2894	2894
q6	238	147	153	147
q7	986	641	606	606
q8	9775	2114	2098	2098
q9	7227	6597	6560	6560
q10	7003	2232	2210	2210
q11	439	243	251	243
q12	399	225	227	225
q13	17767	3059	3012	3012
q14	288	239	233	233
q15	523	492	496	492
q16	491	390	387	387
q17	1010	724	689	689
q18	7523	6851	6769	6769
q19	1387	1049	1050	1049
q20	684	342	321	321
q21	3937	2981	2975	2975
q22	1128	1024	1009	1009
Total cold run time: 110550 ms
Total hot run time: 38466 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4376	4322	4279	4279
q2	369	279	275	275
q3	2939	2634	2708	2634
q4	1907	1633	1638	1633
q5	5459	5445	5696	5445
q6	220	130	131	130
q7	2093	1743	1729	1729
q8	3239	3362	3404	3362
q9	8529	8465	8474	8465
q10	3445	3216	3193	3193
q11	601	501	506	501
q12	784	597	596	596
q13	12537	3021	3045	3021
q14	308	270	291	270
q15	537	485	477	477
q16	478	445	421	421
q17	1785	1474	1500	1474
q18	7823	7417	7520	7417
q19	1671	1479	1587	1479
q20	2101	1854	1838	1838
q21	5517	5302	5331	5302
q22	1125	1009	1038	1009
Total cold run time: 67843 ms
Total hot run time: 54950 ms

doris-robot · 2024-08-27T08:48:12Z

TPC-DS: Total hot run time: 187608 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4378dbf5bc9a276126dc45a34421b713bff0c17, data reload: false
query1	920	376	364	364
query2	6472	1973	1922	1922
query3	6644	224	221	221
query4	33234	23143	23162	23143
query5	4137	494	498	494
query6	262	169	174	169
query7	4581	302	295	295
query8	251	206	207	206
query9	8640	2455	2434	2434
query10	443	298	281	281
query11	15655	14934	14905	14905
query12	151	100	97	97
query13	1614	374	379	374
query14	9782	7264	6807	6807
query15	255	168	171	168
query16	7853	480	467	467
query17	1586	584	561	561
query18	1950	298	310	298
query19	235	157	157	157
query20	122	113	112	112
query21	218	104	103	103
query22	4510	4258	4244	4244
query23	34051	33497	33468	33468
query24	11180	2840	2811	2811
query25	589	391	382	382
query26	1180	181	162	162
query27	2458	292	284	284
query28	7396	2054	2041	2041
query29	796	419	403	403
query30	308	158	148	148
query31	964	772	776	772
query32	95	59	58	58
query33	759	296	292	292
query34	970	485	487	485
query35	852	730	719	719
query36	1082	936	926	926
query37	155	88	84	84
query38	4010	3947	3858	3858
query39	1433	1407	1369	1369
query40	199	116	120	116
query41	47	50	49	49
query42	121	105	100	100
query43	515	478	498	478
query44	1235	749	769	749
query45	193	166	169	166
query46	1102	752	746	746
query47	1889	1751	1813	1751
query48	392	304	293	293
query49	1091	430	429	429
query50	813	445	417	417
query51	7149	7038	7029	7029
query52	100	90	91	90
query53	259	189	183	183
query54	1015	468	457	457
query55	82	79	84	79
query56	322	272	264	264
query57	1196	1071	1067	1067
query58	240	225	267	225
query59	3155	2914	2817	2817
query60	310	277	270	270
query61	113	98	101	98
query62	823	662	662	662
query63	220	185	189	185
query64	5474	2340	1782	1782
query65	3205	3150	3137	3137
query66	1367	337	334	334
query67	15586	15402	15285	15285
query68	3385	596	589	589
query69	402	284	287	284
query70	1131	1101	1150	1101
query71	330	294	284	284
query72	6292	2286	2081	2081
query73	751	330	334	330
query74	9125	9035	8944	8944
query75	3372	2694	2708	2694
query76	1850	1086	1000	1000
query77	472	323	328	323
query78	10913	9374	9974	9374
query79	1107	548	544	544
query80	756	492	517	492
query81	552	231	225	225
query82	709	138	146	138
query83	243	159	156	156
query84	229	81	80	80
query85	692	278	280	278
query86	313	271	260	260
query87	4438	4294	4163	4163
query88	2956	2402	2384	2384
query89	381	297	294	294
query90	2135	210	203	203
query91	134	112	114	112
query92	65	57	54	54
query93	1040	548	536	536
query94	920	314	278	278
query95	435	275	271	271
query96	595	276	276	276
query97	3219	3113	3035	3035
query98	219	210	203	203
query99	1553	1294	1280	1280
Total cold run time: 286720 ms
Total hot run time: 187608 ms

doris-robot · 2024-08-27T08:53:40Z

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4378dbf5bc9a276126dc45a34421b713bff0c17, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.08	0.08
query5	0.52	0.49	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.47	0.47
query10	0.55	0.54	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.61	0.58	0.58
query14	0.77	0.78	0.79
query15	0.90	0.81	0.82
query16	0.37	0.37	0.36
query17	1.07	1.02	1.01
query18	0.20	0.20	0.20
query19	1.88	1.75	1.77
query20	0.01	0.00	0.01
query21	15.42	0.65	0.65
query22	3.61	7.55	1.87
query23	18.26	1.40	1.25
query24	2.16	0.25	0.21
query25	0.14	0.08	0.07
query26	0.27	0.18	0.17
query27	0.08	0.07	0.08
query28	13.16	1.03	1.00
query29	12.67	3.31	3.34
query30	0.24	0.06	0.07
query31	2.85	0.40	0.40
query32	3.26	0.48	0.46
query33	2.95	3.01	3.02
query34	17.00	4.42	4.36
query35	4.47	4.45	4.42
query36	0.66	0.48	0.47
query37	0.19	0.15	0.15
query38	0.16	0.14	0.15
query39	0.05	0.04	0.03
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 108.94 s
Total hot run time: 30.56 s

dataroaring

LGTM

github-actions · 2024-08-27T12:52:28Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-27T12:52:31Z

PR approved by anyone and no changes requested.

liaoxin01

LGTM

…all transactions (#39975) We encountered a scenario where a large number of small transactions were generated, resulting in an impact on query performance: Kafka's data comes in batches of very small data every very short time, which leads to tasks being frequently scheduled and ending very quickly, resulting in a large number of small transactions. To solve this problem, we delay the scheduling of tasks that perceive EOF, which would not delay data consumption, for perceiving EOF indicates that the consumption speed is greater than the production speed.

…all transactions (#39975) (#40498) pick (#39975) We encountered a scenario where a large number of small transactions were generated, resulting in an impact on query performance: Kafka's data comes in batches of very small data every very short time, which leads to tasks being frequently scheduled and ending very quickly, resulting in a large number of small transactions. To solve this problem, we delay the scheduling of tasks that perceive EOF, which would not delay data consumption, for perceiving EOF indicates that the consumption speed is greater than the production speed.

…all transactions (#39975) (#40509) pick (#39975) We encountered a scenario where a large number of small transactions were generated, resulting in an impact on query performance: Kafka's data comes in batches of very small data every very short time, which leads to tasks being frequently scheduled and ending very quickly, resulting in a large number of small transactions. To solve this problem, we delay the scheduling of tasks that perceive EOF, which would not delay data consumption, for perceiving EOF indicates that the consumption speed is greater than the production speed.

Related PR: #39975 In the above PR, we introduce delay schedule EOF tasks to avoid too many small transactions, but it is invalid, the pr fix the issue.

…all transactions (apache#39975) We encountered a scenario where a large number of small transactions were generated, resulting in an impact on query performance: Kafka's data comes in batches of very small data every very short time, which leads to tasks being frequently scheduled and ending very quickly, resulting in a large number of small transactions. To solve this problem, we delay the scheduling of tasks that perceive EOF, which would not delay data consumption, for perceiving EOF indicates that the consumption speed is greater than the production speed.

) Related PR: apache#39975 In the above PR, we introduce delay schedule EOF tasks to avoid too many small transactions, but it is invalid, the pr fix the issue.

sollhui changed the title ~~[fix](routine load) delay schedule EOF tasks to avoid too many small transactions~~[improve](routine load) delay schedule EOF tasks to avoid too many small transactionsAug 27, 2024

delay schedule EOF tasks to avoid too many small transactions
c4378db

dataroaring approved these changes Aug 27, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 27, 2024

dataroaring added dev/2.1.x dev/2.0.x labels Aug 28, 2024

liaoxin01 approved these changes Aug 28, 2024
View reviewed changes

liaoxin01 added the dev/3.0.x label Aug 28, 2024

liaoxin01 merged commit efd1cd0 into apache:masterAug 28, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Sep 3, 2024

This was referenced Sep 7, 2024
[improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) #40498
Merged
[improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) #40509
Merged

dataroaring added dev/2.0.15-merged and removed dev/2.0.x labels Sep 8, 2024

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Sep 9, 2024

xiaokang mentioned this pull request Sep 14, 2024
Release Note 2.0.15 #40882
Closed

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

sollhui mentioned this pull request Dec 3, 2024
[fix](routine load) fix invalid EOF tasks scheduling delay #44949
Merged
16 tasks

Navigation Menu

Uh oh!

[improve](routine load) delay schedule EOF tasks to avoid too many small transactions - #39975

[improve](routine load) delay schedule EOF tasks to avoid too many small transactions#39975
liaoxin01 merged 1 commit into
apache:masterfrom
sollhui:opt_small_txn

sollhui commented Aug 27, 2024•
edited
Loading

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

sollhui commented Aug 27, 2024

Uh oh!

sollhui commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

dataroaring left a comment

Uh oh!

github-actionsBot commented Aug 27, 2024

Uh oh!

github-actionsBot commented Aug 27, 2024

Uh oh!

liaoxin01 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

sollhui commented Aug 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

sollhui commented Aug 27, 2024

Uh oh!

sollhui commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 27, 2024

Uh oh!

github-actionsBot commented Aug 27, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sollhui commented Aug 27, 2024•
edited
Loading