Skip to content

[Fix](load) fix commit txn timeout when loading to table with many tablet - #40031

Merged
BiteTheDDDDt merged 1 commit into
apache:masterfrom
zclllyybb:commit_txn
Aug 29, 2024
Merged

[Fix](load) fix commit txn timeout when loading to table with many tablet#40031
BiteTheDDDDt merged 1 commit into
apache:masterfrom
zclllyybb:commit_txn

Conversation

@zclllyybb

@zclllyybbzclllyybb commented Aug 28, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

#39595 introduces the issue. Wrongly duplicated check partition's partition_number * tablet_number times for all tables when committing load txns.

now only do this check for the table that is restoring. and not check a partition again.

@doris-robot

doris-robot commented Aug 28, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb

zclllyybb commented Aug 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

w41ter
w41ter approved these changes Aug 28, 2024
View reviewed changes

@w41terw41ter left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 28, 2024
@github-actions

github-actionsBot commented Aug 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Aug 28, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 37965 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43f6abac988e8753879f766627ffc008c6ad7116, data reload: false
------ Round 1 ----------------------------------
q1	17628	4383	4277	4277
q2	2022	188	175	175
q3	11980	979	1137	979
q4	10519	745	681	681
q5	7767	2842	2819	2819
q6	225	136	136	136
q7	971	604	597	597
q8	9328	2069	2092	2069
q9	7326	6574	6596	6574
q10	6989	2250	2182	2182
q11	447	252	248	248
q12	404	221	226	221
q13	19082	3024	3056	3024
q14	286	236	237	236
q15	530	501	492	492
q16	583	499	501	499
q17	986	791	674	674
q18	7263	6896	6797	6797
q19	1398	983	1107	983
q20	690	342	353	342
q21	3987	2944	3105	2944
q22	1111	1016	1019	1016
Total cold run time: 111522 ms
Total hot run time: 37965 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4373	4276	4318	4276
q2	385	278	274	274
q3	2869	2696	2671	2671
q4	1964	1638	1679	1638
q5	5658	5678	5757	5678
q6	239	143	140	140
q7	2227	1816	1847	1816
q8	3350	3443	3448	3443
q9	8908	8890	8870	8870
q10	3603	3412	3387	3387
q11	620	527	516	516
q12	838	646	665	646
q13	15532	3185	3270	3185
q14	331	293	302	293
q15	536	492	507	492
q16	623	567	576	567
q17	1870	1584	1573	1573
q18	8156	7828	7923	7828
q19	1737	1468	1462	1462
q20	2152	1911	1900	1900
q21	5806	5592	5532	5532
q22	1146	1062	1054	1054
Total cold run time: 72923 ms
Total hot run time: 57241 ms

@doris-robot

doris-robot commented Aug 28, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193669 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43f6abac988e8753879f766627ffc008c6ad7116, data reload: false
query1	1249	875	862	862
query2	6323	2033	1981	1981
query3	10709	4195	4051	4051
query4	60143	26631	23311	23311
query5	5410	497	510	497
query6	431	166	191	166
query7	5826	294	293	293
query8	283	207	202	202
query9	9046	2510	2481	2481
query10	474	270	263	263
query11	17989	15291	15472	15291
query12	159	98	101	98
query13	1543	382	380	380
query14	10894	7680	7265	7265
query15	247	180	180	180
query16	7471	476	490	476
query17	1135	571	578	571
query18	2025	298	306	298
query19	278	146	146	146
query20	116	109	109	109
query21	224	102	107	102
query22	4523	4382	4381	4381
query23	34455	33557	33451	33451
query24	5963	2865	2852	2852
query25	548	392	401	392
query26	695	160	158	158
query27	1779	297	285	285
query28	3965	2125	2113	2113
query29	711	429	428	428
query30	242	158	158	158
query31	891	770	782	770
query32	84	57	67	57
query33	469	327	284	284
query34	857	494	474	474
query35	824	720	733	720
query36	1050	918	941	918
query37	157	96	93	93
query38	3955	3850	3901	3850
query39	1416	1411	1439	1411
query40	191	119	116	116
query41	48	42	45	42
query42	112	97	97	97
query43	517	495	473	473
query44	1116	756	753	753
query45	194	167	171	167
query46	1089	738	752	738
query47	1892	1805	1793	1793
query48	394	287	297	287
query49	755	423	445	423
query50	812	412	404	404
query51	7197	7054	7113	7054
query52	101	87	114	87
query53	251	181	186	181
query54	561	454	447	447
query55	74	76	79	76
query56	269	252	262	252
query57	1205	1110	1073	1073
query58	229	228	220	220
query59	3229	2843	2713	2713
query60	288	279	282	279
query61	105	101	96	96
query62	735	656	662	656
query63	214	184	186	184
query64	2836	686	686	686
query65	3238	3123	3161	3123
query66	623	332	353	332
query67	15490	15224	15332	15224
query68	2963	597	576	576
query69	416	286	285	285
query70	1175	1155	1121	1121
query71	337	276	272	272
query72	5278	4081	4052	4052
query73	756	336	330	330
query74	9116	8807	8795	8795
query75	3347	2662	2691	2662
query76	1376	1042	997	997
query77	532	328	328	328
query78	10891	9228	9110	9110
query79	1064	547	540	540
query80	685	509	520	509
query81	478	237	235	235
query82	243	148	151	148
query83	176	148	148	148
query84	260	78	77	77
query85	681	358	280	280
query86	308	293	285	285
query87	4374	4370	4247	4247
query88	2938	2478	2341	2341
query89	373	294	298	294
query90	2058	195	194	194
query91	123	100	100	100
query92	60	53	50	50
query93	1041	532	534	532
query94	709	309	298	298
query95	340	261	265	261
query96	589	270	269	269
query97	3212	3081	3085	3081
query98	217	206	215	206
query99	1519	1291	1252	1252
Total cold run time: 305959 ms
Total hot run time: 193669 ms

@doris-robot

doris-robot commented Aug 28, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43f6abac988e8753879f766627ffc008c6ad7116, data reload: false
query1	0.04	0.04	0.05
query2	0.09	0.04	0.05
query3	0.23	0.05	0.05
query4	1.66	0.08	0.08
query5	0.51	0.50	0.50
query6	1.14	0.74	0.73
query7	0.01	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.50	0.49
query10	0.56	0.55	0.54
query11	0.17	0.12	0.12
query12	0.16	0.12	0.13
query13	0.63	0.59	0.60
query14	2.03	2.04	2.12
query15	0.85	0.84	0.83
query16	0.36	0.37	0.36
query17	0.99	1.02	0.99
query18	0.21	0.20	0.22
query19	1.86	1.78	1.74
query20	0.02	0.01	0.01
query21	15.39	0.68	0.66
query22	4.09	7.20	2.63
query23	18.31	1.29	1.27
query24	2.12	0.22	0.22
query25	0.15	0.08	0.08
query26	0.29	0.19	0.17
query27	0.08	0.08	0.08
query28	13.22	1.02	1.01
query29	12.86	3.34	3.34
query30	0.24	0.06	0.06
query31	2.89	0.40	0.41
query32	3.22	0.48	0.48
query33	2.96	2.96	2.99
query34	16.85	4.43	4.40
query35	4.51	4.57	4.46
query36	0.65	0.47	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.18	0.13	0.13
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.73 s
Total hot run time: 32.82 s

@BiteTheDDDDt
BiteTheDDDDt merged commit 83e6eac into apache:masterAug 29, 2024
@zclllyybb
zclllyybb deleted the commit_txn branch August 29, 2024 02:57
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Aug 29, 2024
@zclllyybb
[Fix](load) fix commit txn timeout when loading to table with many ta…
28620aa
…blet (apache#40031)
fix commit txn timeout when loading to table with many tablet
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Aug 29, 2024
@zclllyybb
[Fix](load) fix commit txn timeout when loading to table with many ta…
275f6e4
…blet (apache#40031)
fix commit txn timeout when loading to table with many tablet
This was referenced Aug 29, 2024
Merged
Merged
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Aug 29, 2024
@zclllyybb
[Fix](load) fix commit txn timeout when loading to table with many ta…
260eb65
…blet (apache#40031)
fix commit txn timeout when loading to table with many tablet
yiguolei pushed a commit to zclllyybb/doris that referenced this pull request Aug 29, 2024
@zclllyybb@yiguolei
[Fix](load) fix commit txn timeout when loading to table with many ta…
698e624
…blet (apache#40031)
fix commit txn timeout when loading to table with many tablet
yiguolei pushed a commit that referenced this pull request Aug 30, 2024
@zclllyybb
[branch-2.1](load) fix commit txn timeout when loading to table with …
e4d543f
…many tablet (#40031) (#40139)
pick #40031
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@zclllyybb@dataroaring
[Fix](load) fix commit txn timeout when loading to table with many ta…
c0c17b3
…blet (#40031)
fix commit txn timeout when loading to table with many tablet
xiaokang pushed a commit that referenced this pull request Sep 11, 2024
@zclllyybb
[fix](load) fix commit txn timeout when loading to table with many ta…
1617866
…blet (#40031) (#40140)
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@zclllyybb@HappenLee
[Fix](load) fix commit txn timeout when loading to table with many ta…
6545236
…blet (apache#40031)
fix commit txn timeout when loading to table with many tablet
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@w41terw41terw41ter approved these changes
@BiteTheDDDDtBiteTheDDDDtBiteTheDDDDt approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.15-mergeddev/2.1.6-mergeddev/3.0.2-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@zclllyybb@doris-robot@w41ter@BiteTheDDDDt@xiaokang@yiguolei@zhangstar333@dataroaring