[improve](txn insert) txn insert support write to one table many times by mymeiyi · Pull Request #32980 · apache/doris

mymeiyi · 2024-03-28T11:22:55Z

Proposed changes

Purpose

We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load.

But leave a problem that, one partition can only be written once in one transaction, because current transaction mechanism only support publish one version for one partition. This pr is to solve this problem.

In other words, this pr supports write to one table many times in one transaction like:

begin;
insert into t1 select ...;
insert into t1 select ...;
commit;

Current implementation

In Doris, one transaction is related to one txn_id
BE use this txn_id to record the load info of partition_id, tablet, DeltaWriter... in txn_manager
If writing to one partition twice in one txn, the above info in BE may be overwrited
When FE commit the txn, it calcultes a new partition version, a version is related to a Rowset, but multiple loads in txn generate multiple Rowsets.

New implementation

Introduce of sub_txn_id

To solve the above problem, the basic idea is to separate the txn_id in FE and BE. For multiple loads in one txn, we use sub_txn_id to distinguish the load for BE.

One example: suppose table t has 2 partitions, p1 and p2. The current version of p1 is 3, p2 is 4.

command	FE	BE
BEGIN	mark txn is began
insert into t partition(p1, p2)	1. begin_txn -> txn_id; 2. sub_txn_id1 = txn_id;	load use sub_txn_id1
insert into t partition(p1)	generate sub_txn_id2	load use sub_txn_id2
insert into t partition(p1, p2)	generate sub_txn_id3	load use sub_txn_id3
COMMIT	commit_txn(calculate partition version): * sub_txn_id1: p1(4), p2(5) * sub_txn_id2: p1(5) * sub_txn_id3: p1(6), p2(6) publish_task: * use sub_txn_id to submit publish version tasks to be

FE Meta

In addition, this pr change the storage of TransationState in bdbje to json format to make it compatible.

Isolation Level

Doris provides the READ COMMITTED isolation level. Please note the following:

In a transaction, each statement reads the data that was committed at the time the statement began executing.
In a transaction, each statement cannot read the modifications made by other statements within the same transaction. Please notice:
1. For delete command, there are 2 implementations, one is delete condition, one is insert.
2. If the delete condition is committed after the insert, the delete will work for the insert, for example:
```
begin;
insert into t select * from t1 where id = 1;
delete from t where id = 1;
commit;
```
1. So we make the delete condition commit before the insert.

User doc

apache/doris-website#604

doris-robot · 2024-03-28T11:23:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

mymeiyi · 2024-03-28T11:25:45Z

run buildall

mymeiyi · 2024-03-28T12:43:39Z

run buildall

doris-robot · 2024-03-28T13:19:23Z

TPC-H: Total hot run time: 37510 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2b70a5247bbccb71d9a68f6ac8e657ad5b9c27b, data reload: false
------ Round 1 ----------------------------------
q1	17608	4319	4104	4104
q2	2105	157	166	157
q3	10563	1151	1167	1151
q4	10229	834	817	817
q5	7451	2892	2892	2892
q6	205	126	124	124
q7	1015	586	576	576
q8	9333	1960	1948	1948
q9	6952	6361	6282	6282
q10	8435	3460	3611	3460
q11	445	227	217	217
q12	403	197	197	197
q13	17790	2834	2846	2834
q14	244	199	204	199
q15	517	464	460	460
q16	483	366	363	363
q17	955	619	529	529
q18	7093	6482	6378	6378
q19	4370	1427	1458	1427
q20	547	264	265	264
q21	3566	2871	2839	2839
q22	336	294	292	292
Total cold run time: 110645 ms
Total hot run time: 37510 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4125	4121	4070	4070
q2	336	227	233	227
q3	2942	2794	2825	2794
q4	1827	1566	1525	1525
q5	5193	5227	5246	5227
q6	196	120	121	120
q7	2228	1804	1830	1804
q8	3159	3313	3293	3293
q9	8438	8358	8408	8358
q10	3776	3909	3962	3909
q11	552	461	460	460
q12	759	585	584	584
q13	17076	3039	3080	3039
q14	313	260	280	260
q15	513	473	475	473
q16	510	472	477	472
q17	1810	1560	1533	1533
q18	7762	7595	7514	7514
q19	1660	1551	1602	1551
q20	2048	1798	1794	1794
q21	5074	4786	4903	4786
q22	540	485	457	457
Total cold run time: 70837 ms
Total hot run time: 54250 ms

doris-robot · 2024-03-28T13:31:16Z

TPC-DS: Total hot run time: 181883 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f2b70a5247bbccb71d9a68f6ac8e657ad5b9c27b, data reload: false
query1	926	363	369	363
query2	6379	2017	1997	1997
query3	6706	205	201	201
query4	31760	21338	21296	21296
query5	4325	406	390	390
query6	274	197	181	181
query7	4625	296	297	296
query8	236	195	170	170
query9	9371	2306	2271	2271
query10	438	256	269	256
query11	14796	14287	14237	14237
query12	138	94	86	86
query13	1619	425	428	425
query14	10344	7671	7986	7671
query15	252	181	182	181
query16	8173	240	248	240
query17	1973	558	525	525
query18	2084	277	277	277
query19	321	152	144	144
query20	94	84	81	81
query21	206	134	131	131
query22	5018	4845	4703	4703
query23	33594	32903	33268	32903
query24	10847	2937	2990	2937
query25	580	400	409	400
query26	708	175	165	165
query27	2292	361	362	361
query28	6194	1900	1974	1900
query29	929	682	647	647
query30	306	153	157	153
query31	1017	808	757	757
query32	95	60	58	58
query33	669	254	260	254
query34	1030	520	508	508
query35	902	750	704	704
query36	1036	926	910	910
query37	105	67	68	67
query38	3828	3686	3628	3628
query39	1121	1055	1057	1055
query40	179	112	113	112
query41	51	48	45	45
query42	109	102	98	98
query43	494	467	475	467
query44	1159	771	756	756
query45	289	260	271	260
query46	1117	703	724	703
query47	1988	1939	1948	1939
query48	447	375	369	369
query49	876	357	340	340
query50	789	405	401	401
query51	6862	6841	6815	6815
query52	126	98	90	90
query53	353	309	284	284
query54	308	254	247	247
query55	88	82	100	82
query56	248	232	233	232
query57	1251	1199	1182	1182
query58	227	212	212	212
query59	3022	2681	2622	2622
query60	248	232	237	232
query61	95	89	90	89
query62	606	468	439	439
query63	303	288	280	280
query64	4915	3989	3884	3884
query65	3076	3066	3051	3051
query66	785	363	358	358
query67	16509	14881	14859	14859
query68	8583	533	544	533
query69	642	396	376	376
query70	1227	1212	1194	1194
query71	518	262	276	262
query72	6321	2547	2348	2348
query73	938	307	316	307
query74	8009	6338	6359	6338
query75	3731	2173	2207	2173
query76	5250	858	887	858
query77	613	258	252	252
query78	10877	10234	10034	10034
query79	12893	520	511	511
query80	2000	377	377	377
query81	534	217	212	212
query82	760	85	79	79
query83	202	141	137	137
query84	286	80	75	75
query85	1259	329	306	306
query86	395	310	288	288
query87	3745	3509	3476	3476
query88	5372	2328	2318	2318
query89	508	367	366	366
query90	1977	172	178	172
query91	175	132	144	132
query92	57	45	46	45
query93	7103	482	475	475
query94	1174	179	172	172
query95	404	315	303	303
query96	591	272	276	272
query97	2619	2488	2486	2486
query98	252	217	210	210
query99	1150	858	819	819
Total cold run time: 311996 ms
Total hot run time: 181883 ms

doris-robot · 2024-03-28T13:39:39Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f2b70a5247bbccb71d9a68f6ac8e657ad5b9c27b with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.8 seconds inserted 10000000 Rows, about 724K ops/s

mymeiyi · 2024-03-29T03:29:15Z

run buildall

doris-robot · 2024-03-29T04:48:17Z

TPC-H: Total hot run time: 37824 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf, data reload: false
------ Round 1 ----------------------------------
q1	17630	4225	4130	4130
q2	2126	163	155	155
q3	10583	1179	1169	1169
q4	10224	744	798	744
q5	7474	2981	2936	2936
q6	207	124	124	124
q7	1034	589	575	575
q8	9336	2024	1992	1992
q9	7224	6336	6341	6336
q10	8441	3461	3577	3461
q11	422	232	219	219
q12	370	198	192	192
q13	17801	2846	2845	2845
q14	233	210	200	200
q15	496	470	461	461
q16	466	370	368	368
q17	956	567	647	567
q18	7311	6545	6527	6527
q19	2708	1439	1515	1439
q20	556	254	257	254
q21	3602	2929	2840	2840
q22	347	290	307	290
Total cold run time: 109547 ms
Total hot run time: 37824 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4102	4067	4082	4067
q2	328	228	243	228
q3	2966	2852	2819	2819
q4	1891	1606	1599	1599
q5	5223	5246	5243	5243
q6	194	117	115	115
q7	2216	1804	1798	1798
q8	3177	3286	3273	3273
q9	8434	8384	8370	8370
q10	3834	3891	3949	3891
q11	569	462	459	459
q12	754	552	603	552
q13	16954	3095	3090	3090
q14	304	283	260	260
q15	538	497	492	492
q16	516	473	463	463
q17	1796	1532	1528	1528
q18	7928	7776	7475	7475
q19	3488	1539	1584	1539
q20	2010	1819	1872	1819
q21	4969	4948	4784	4784
q22	562	443	449	443
Total cold run time: 72753 ms
Total hot run time: 54307 ms

doris-robot · 2024-03-29T05:00:19Z

TPC-DS: Total hot run time: 182424 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf, data reload: false
query1	920	366	355	355
query2	6369	2052	1989	1989
query3	6703	225	209	209
query4	31822	21232	21263	21232
query5	4332	417	413	413
query6	281	187	182	182
query7	4640	288	293	288
query8	234	174	176	174
query9	9199	2266	2248	2248
query10	445	257	268	257
query11	17266	14199	14298	14199
query12	142	86	89	86
query13	1622	427	420	420
query14	10151	7862	7221	7221
query15	240	186	181	181
query16	8197	269	273	269
query17	1923	584	552	552
query18	2113	316	293	293
query19	273	159	168	159
query20	102	95	88	88
query21	204	131	126	126
query22	5023	4856	4756	4756
query23	33410	32996	33191	32996
query24	12208	2964	2964	2964
query25	660	413	411	411
query26	1646	177	167	167
query27	3198	372	372	372
query28	7889	1921	1922	1921
query29	1005	642	647	642
query30	320	157	156	156
query31	1029	776	782	776
query32	99	61	59	59
query33	778	272	264	264
query34	1334	503	518	503
query35	937	733	747	733
query36	1067	920	924	920
query37	266	69	66	66
query38	3774	3704	3613	3613
query39	1655	1630	1592	1592
query40	257	116	119	116
query41	53	49	49	49
query42	111	98	104	98
query43	490	472	454	454
query44	1187	759	760	759
query45	297	256	267	256
query46	1107	719	709	709
query47	2042	1935	1920	1920
query48	475	373	376	373
query49	1031	369	376	369
query50	788	396	394	394
query51	6912	6842	6835	6835
query52	105	95	95	95
query53	344	283	288	283
query54	307	240	265	240
query55	86	84	86	84
query56	254	234	241	234
query57	1249	1258	1213	1213
query58	222	227	233	227
query59	2893	2640	2443	2443
query60	255	245	238	238
query61	96	95	93	93
query62	607	437	435	435
query63	303	272	278	272
query64	6253	4119	3934	3934
query65	3076	3024	3014	3014
query66	1321	348	351	348
query67	15539	14917	14747	14747
query68	9270	550	540	540
query69	672	377	385	377
query70	1451	1119	1161	1119
query71	525	267	273	267
query72	6795	2521	2358	2358
query73	1654	327	321	321
query74	6912	6404	6400	6400
query75	3904	2207	2195	2195
query76	5464	843	901	843
query77	603	260	264	260
query78	10958	10151	10181	10151
query79	11586	538	540	538
query80	1777	376	378	376
query81	497	221	223	221
query82	238	82	88	82
query83	201	148	155	148
query84	290	80	80	80
query85	1045	312	316	312
query86	351	301	287	287
query87	3836	3562	3546	3546
query88	5476	2377	2364	2364
query89	487	379	370	370
query90	2064	178	176	176
query91	173	136	137	136
query92	63	47	47	47
query93	6230	497	504	497
query94	1628	190	181	181
query95	408	317	316	316
query96	605	267	271	267
query97	2668	2496	2513	2496
query98	233	214	215	214
query99	1035	855	859	855
Total cold run time: 320235 ms
Total hot run time: 182424 ms

doris-robot · 2024-03-29T05:05:37Z

ClickBench: Total hot run time: 28.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf, data reload: false
query1	0.04	0.04	0.04
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.64	0.10	0.09
query5	0.51	0.48	0.50
query6	1.15	0.63	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.48	0.49
query10	0.54	0.56	0.56
query11	0.13	0.09	0.09
query12	0.12	0.10	0.09
query13	0.58	0.58	0.57
query14	0.77	0.77	0.74
query15	0.83	0.80	0.79
query16	0.39	0.37	0.38
query17	1.00	0.99	0.95
query18	0.23	0.24	0.22
query19	1.78	1.66	1.74
query20	0.01	0.00	0.01
query21	15.56	0.56	0.54
query22	2.79	2.89	1.27
query23	17.24	0.72	0.75
query24	3.50	1.08	1.08
query25	0.14	0.22	0.10
query26	0.56	0.13	0.12
query27	0.03	0.03	0.07
query28	11.07	0.84	0.87
query29	12.51	3.25	3.14
query30	0.28	0.09	0.08
query31	2.83	0.35	0.36
query32	3.34	0.44	0.44
query33	2.78	2.84	2.76
query34	16.09	4.35	4.44
query35	4.49	4.41	4.42
query36	0.64	0.48	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.91 s
Total hot run time: 28.9 s

doris-robot · 2024-03-29T05:08:43Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c9064b19b33ba6d3d773f219b495c028707f82bf with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s

dataroaring

one comment.

mymeiyi · 2024-04-01T08:21:51Z

run buildall

mymeiyi · 2024-04-07T11:26:31Z

run buildall

doris-robot · 2024-04-07T12:01:44Z

TPC-H: Total hot run time: 38520 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d, data reload: false
------ Round 1 ----------------------------------
q1	17615	4135	4067	4067
q2	2023	191	185	185
q3	10564	1179	1326	1179
q4	10242	804	974	804
q5	7486	2937	2915	2915
q6	220	137	135	135
q7	1118	625	615	615
q8	9409	1983	2016	1983
q9	6621	6179	6153	6153
q10	8441	3490	3496	3490
q11	417	248	244	244
q12	380	211	213	211
q13	17783	2905	2913	2905
q14	265	236	236	236
q15	524	485	488	485
q16	517	375	373	373
q17	941	886	889	886
q18	7201	6539	6408	6408
q19	1588	1522	1520	1520
q20	539	311	310	310
q21	3484	3131	3118	3118
q22	361	298	314	298
Total cold run time: 107739 ms
Total hot run time: 38520 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4069	4040	4059	4040
q2	325	220	221	220
q3	2968	2953	2961	2953
q4	1853	1832	1825	1825
q5	5231	5227	5227	5227
q6	210	127	127	127
q7	2265	1817	1764	1764
q8	3226	3300	3284	3284
q9	8457	8501	8508	8501
q10	3767	3948	4009	3948
q11	562	498	454	454
q12	782	576	609	576
q13	17162	3127	3137	3127
q14	307	283	279	279
q15	523	473	495	473
q16	494	454	449	449
q17	1786	1758	1757	1757
q18	8311	7736	7821	7736
q19	1711	1690	1693	1690
q20	2040	1825	1876	1825
q21	5216	5048	5009	5009
q22	494	444	433	433
Total cold run time: 71759 ms
Total hot run time: 55697 ms

doris-robot · 2024-04-07T12:13:20Z

TPC-DS: Total hot run time: 181653 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d, data reload: false
query1	905	1112	1110	1110
query2	6305	2041	1950	1950
query3	8266	226	221	221
query4	26608	21685	21300	21300
query5	4182	401	399	399
query6	285	191	191	191
query7	4603	312	291	291
query8	239	187	171	171
query9	8498	2202	2222	2202
query10	464	245	251	245
query11	14882	14453	14406	14406
query12	139	92	86	86
query13	1633	370	384	370
query14	9352	6996	6939	6939
query15	219	180	178	178
query16	7119	268	270	268
query17	1711	563	551	551
query18	1817	268	271	268
query19	203	154	148	148
query20	98	86	85	85
query21	195	123	124	123
query22	5005	4811	4778	4778
query23	33020	32629	32556	32556
query24	9984	3184	3156	3156
query25	594	383	390	383
query26	723	158	169	158
query27	3019	364	392	364
query28	5741	1923	1867	1867
query29	953	620	617	617
query30	340	177	175	175
query31	1004	780	796	780
query32	101	55	58	55
query33	573	262	257	257
query34	905	508	521	508
query35	828	720	696	696
query36	1018	904	879	879
query37	106	75	77	75
query38	3737	3617	3610	3610
query39	1624	1604	1593	1593
query40	176	130	128	128
query41	53	46	46	46
query42	109	102	108	102
query43	467	441	429	429
query44	1107	763	732	732
query45	288	264	251	251
query46	1090	821	776	776
query47	1977	1863	1889	1863
query48	397	314	316	314
query49	833	371	369	369
query50	830	419	414	414
query51	6841	6738	6768	6738
query52	108	93	95	93
query53	357	291	299	291
query54	266	233	232	232
query55	87	77	76	76
query56	245	240	248	240
query57	1261	1167	1188	1167
query58	232	219	227	219
query59	2714	2429	2379	2379
query60	249	258	243	243
query61	111	115	87	87
query62	663	456	450	450
query63	304	277	279	277
query64	4248	3150	3125	3125
query65	3045	3025	2980	2980
query66	735	322	308	308
query67	15703	14883	14741	14741
query68	8945	578	583	578
query69	552	295	304	295
query70	1315	1120	1090	1090
query71	487	261	272	261
query72	6269	2568	2441	2441
query73	970	319	325	319
query74	6693	6291	6357	6291
query75	3663	2300	2299	2299
query76	5546	1208	1226	1208
query77	650	244	255	244
query78	10838	10150	10147	10147
query79	9946	521	524	521
query80	1872	419	419	419
query81	496	237	232	232
query82	528	98	90	90
query83	210	165	163	163
query84	273	86	87	86
query85	1055	274	270	270
query86	371	293	286	286
query87	3654	3490	3517	3490
query88	3724	2346	2353	2346
query89	565	374	366	366
query90	1943	180	177	177
query91	132	105	101	101
query92	59	48	48	48
query93	6651	521	516	516
query94	1118	177	181	177
query95	418	311	308	308
query96	607	270	268	268
query97	2621	2499	2480	2480
query98	231	216	225	216
query99	1297	859	841	841
Total cold run time: 295193 ms
Total hot run time: 181653 ms

doris-robot · 2024-04-07T12:18:47Z

ClickBench: Total hot run time: 30.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.04	0.04
query4	1.67	0.06	0.06
query5	0.49	0.48	0.49
query6	1.14	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.55	0.49	0.50
query10	0.57	0.57	0.56
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.60	0.60
query14	0.78	0.80	0.78
query15	0.86	0.85	0.85
query16	0.34	0.35	0.36
query17	0.96	0.97	0.96
query18	0.25	0.24	0.25
query19	1.84	1.72	1.75
query20	0.01	0.00	0.01
query21	15.42	0.65	0.64
query22	3.81	6.66	1.83
query23	18.04	1.37	1.26
query24	1.51	0.20	0.19
query25	0.16	0.07	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.85	0.97	0.94
query29	12.59	3.26	3.27
query30	0.26	0.06	0.05
query31	2.88	0.40	0.39
query32	3.24	0.48	0.48
query33	2.86	2.91	2.88
query34	15.51	4.34	4.33
query35	4.38	4.34	4.36
query36	0.68	0.48	0.47
query37	0.18	0.16	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.18	0.14	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 107.06 s
Total hot run time: 30.1 s

doris-robot · 2024-04-07T12:21:53Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 729cc3012a86c45146c309916d4ad1538470cc2d with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 16.7 seconds inserted 10000000 Rows, about 598K ops/s

mymeiyi · 2024-04-08T03:49:01Z

run buildall

doris-robot · 2024-04-08T04:07:00Z

TPC-H: Total hot run time: 38735 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ed7e2d1f979eda02f060b4ccb98d95d39336544, data reload: false
------ Round 1 ----------------------------------
q1	18399	4495	4390	4390
q2	2636	189	189	189
q3	11164	1216	1213	1213
q4	10375	771	726	726
q5	7500	2744	2644	2644
q6	217	135	135	135
q7	1003	596	586	586
q8	9221	2085	2065	2065
q9	7915	6602	6548	6548
q10	8512	3591	3528	3528
q11	463	232	225	225
q12	384	220	208	208
q13	18713	2980	2964	2964
q14	268	242	243	242
q15	511	479	472	472
q16	523	402	374	374
q17	972	718	698	698
q18	7437	6700	6741	6700
q19	1588	1506	1497	1497
q20	690	300	312	300
q21	3316	2736	2750	2736
q22	360	295	305	295
Total cold run time: 112167 ms
Total hot run time: 38735 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4253	4184	4226	4184
q2	371	263	256	256
q3	3005	2756	2719	2719
q4	1920	1636	1609	1609
q5	5289	5298	5306	5298
q6	212	124	125	124
q7	2219	1865	1865	1865
q8	3188	3351	3315	3315
q9	8665	8632	8594	8594
q10	3937	3706	3647	3647
q11	579	480	481	480
q12	726	573	563	563
q13	17537	2966	2933	2933
q14	279	270	269	269
q15	502	461	471	461
q16	480	424	421	421
q17	1782	1491	1471	1471
q18	7482	7456	7320	7320
q19	1625	1543	1581	1543
q20	1975	1739	1740	1739
q21	4958	4697	4723	4697
q22	529	450	457	450
Total cold run time: 71513 ms
Total hot run time: 53958 ms

doris-robot · 2024-04-08T04:18:52Z

TPC-DS: Total hot run time: 181749 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ed7e2d1f979eda02f060b4ccb98d95d39336544, data reload: false
query1	888	1111	1109	1109
query2	6464	1883	1868	1868
query3	6654	202	212	202
query4	37546	21732	21454	21454
query5	4137	395	391	391
query6	223	176	175	175
query7	4044	286	283	283
query8	218	165	167	165
query9	5751	2283	2273	2273
query10	510	236	242	236
query11	14572	14211	14151	14151
query12	138	91	88	88
query13	979	354	353	353
query14	10275	6754	6809	6754
query15	207	184	177	177
query16	6749	261	263	261
query17	1673	595	557	557
query18	1397	291	271	271
query19	194	153	148	148
query20	91	88	82	82
query21	199	131	132	131
query22	4985	4899	4849	4849
query23	33569	33017	32992	32992
query24	12254	2999	2930	2930
query25	541	358	377	358
query26	1849	150	158	150
query27	3085	301	318	301
query28	7419	1912	1878	1878
query29	855	590	584	584
query30	306	162	163	162
query31	887	677	711	677
query32	59	52	51	51
query33	584	247	245	245
query34	888	468	476	468
query35	816	685	700	685
query36	1038	924	898	898
query37	230	69	71	69
query38	3551	3388	3417	3388
query39	1579	1533	1553	1533
query40	276	131	138	131
query41	50	47	46	46
query42	106	98	98	98
query43	503	436	456	436
query44	1445	717	720	717
query45	280	264	275	264
query46	1083	716	733	716
query47	1903	1871	1833	1833
query48	375	290	289	289
query49	1150	372	364	364
query50	761	383	393	383
query51	6824	6576	6696	6576
query52	106	93	96	93
query53	356	280	282	280
query54	265	234	230	230
query55	78	73	74	73
query56	241	222	227	222
query57	1216	1131	1151	1131
query58	230	204	208	204
query59	2845	2710	2559	2559
query60	259	241	252	241
query61	107	111	109	109
query62	624	434	422	422
query63	313	284	286	284
query64	4996	4145	3964	3964
query65	3153	3063	3029	3029
query66	1345	340	328	328
query67	15431	14963	15219	14963
query68	8461	537	532	532
query69	607	311	313	311
query70	1254	1150	1127	1127
query71	548	272	267	267
query72	6621	2741	2438	2438
query73	833	317	316	316
query74	7109	6383	6347	6347
query75	3686	2283	2354	2283
query76	5007	1100	1100	1100
query77	647	238	238	238
query78	10869	10260	10187	10187
query79	9961	535	510	510
query80	2055	413	418	413
query81	502	230	234	230
query82	1313	93	98	93
query83	216	170	167	167
query84	263	79	82	79
query85	1111	256	258	256
query86	500	291	280	280
query87	3729	3556	3537	3537
query88	6119	2287	2292	2287
query89	533	366	376	366
query90	1981	182	181	181
query91	120	96	95	95
query92	61	46	49	46
query93	7212	507	497	497
query94	1214	176	177	176
query95	397	304	310	304
query96	600	265	261	261
query97	2678	2474	2498	2474
query98	241	219	210	210
query99	1167	863	809	809
Total cold run time: 311340 ms
Total hot run time: 181749 ms

doris-robot · 2024-04-08T04:24:20Z

ClickBench: Total hot run time: 31.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ed7e2d1f979eda02f060b4ccb98d95d39336544, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.49	0.50
query6	1.11	0.66	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.51	0.50
query10	0.55	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	0.77	0.78	0.81
query15	0.85	0.83	0.83
query16	0.38	0.38	0.37
query17	1.01	0.94	0.94
query18	0.20	0.26	0.23
query19	1.85	1.71	1.87
query20	0.02	0.01	0.01
query21	15.42	0.65	0.65
query22	4.52	6.33	2.66
query23	18.27	1.23	1.15
query24	1.66	0.31	0.21
query25	0.12	0.08	0.08
query26	0.26	0.15	0.15
query27	0.08	0.08	0.07
query28	13.32	1.02	1.00
query29	12.62	3.32	3.33
query30	0.25	0.06	0.06
query31	2.85	0.40	0.38
query32	3.30	0.47	0.47
query33	2.93	2.86	2.85
query34	17.07	4.50	4.44
query35	4.55	4.48	4.55
query36	0.65	0.47	0.48
query37	0.20	0.16	0.14
query38	0.15	0.15	0.14
query39	0.06	0.04	0.03
query40	0.19	0.14	0.14
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.47 s
Total hot run time: 31.14 s

mymeiyi · 2024-05-07T01:57:42Z

run p0

morningman

LGTM

github-actions · 2024-05-07T03:11:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-07T03:11:39Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

mymeiyi · 2024-05-07T08:32:44Z

run buildall

mymeiyi · 2024-05-07T10:39:10Z

run cloud_p1

dataroaring

LGTM

apache#32980)

## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`

…isible (#39786) introduced in #32980 if a txn is committed twice, the check txn state is committed or visible and skip commit logic does not work

apache#32980)

## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(apache#31666), update(apache#33034) and delete(apache#33100) in transaction load. apache#32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of apache#32980 ### Implementation #### sub_txn_id see apache#32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`

…isible (apache#39786) introduced in apache#32980 if a txn is committed twice, the check txn state is committed or visible and skip commit logic does not work

mymeiyi force-pushed the txn-insert-7 branch from bde92ed to f2b70a5Compare March 28, 2024 12:43

mymeiyi force-pushed the txn-insert-7 branch from f2b70a5 to c9064b1Compare March 29, 2024 03:26

dataroaring reviewed Apr 1, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java

dataroaring reviewed Apr 1, 2024
View reviewed changes

mymeiyi force-pushed the txn-insert-7 branch from c9064b1 to db36fe0Compare April 1, 2024 08:21

mymeiyi force-pushed the txn-insert-7 branch from db36fe0 to 4aa4514Compare April 7, 2024 11:20

github-actionsBot added the meta-change label Apr 7, 2024

mymeiyi force-pushed the txn-insert-7 branch from 4aa4514 to 729cc30Compare April 7, 2024 11:23

mymeiyi force-pushed the txn-insert-7 branch from 729cc30 to 8ed7e2dCompare April 8, 2024 03:47

morningman approved these changes May 7, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 7, 2024

github-actionsBot added the reviewed label May 7, 2024

dataroaring approved these changes May 7, 2024
View reviewed changes

mymeiyi force-pushed the txn-insert-7 branch from f41f417 to c6ee37eCompare May 7, 2024 06:14

mymeiyi added 6 commits May 7, 2024 16:32

[improve](txn insert) txn insert support write to one table many times
318851e

fix check-style
e602bee

fix conflict
4552b26

remove
ed46b20

add timeout
bd266f5

modify serialize name
604b176

mymeiyi force-pushed the txn-insert-7 branch from c6ee37e to 604b176Compare May 7, 2024 08:32

dataroaring approved these changes May 7, 2024
View reviewed changes

dataroaring merged commit 42c4fad into apache:masterMay 7, 2024

ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
[improve](txn insert) txn insert support write to one table many times (
a877143
apache#32980)

mymeiyi mentioned this pull request May 25, 2024
[improve](txn insert) Txn load support cloud mode #34721
Merged

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

dataroaring mentioned this pull request Aug 23, 2024
[fix](commit) Fix does not skip commit if txn state is committed or visible #39786
Merged

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[improve](txn insert) txn insert support write to one table many times (
142c807
apache#32980)

Uh oh!

Conversation

mymeiyi commented Mar 28, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Purpose

Current implementation

New implementation

Introduce of sub_txn_id

FE Meta

Isolation Level

User doc

Uh oh!

doris-robot commented Mar 28, 2024

Uh oh!

mymeiyi commented Mar 28, 2024

Uh oh!

mymeiyi commented Mar 28, 2024

Uh oh!

doris-robot commented Mar 28, 2024

Uh oh!

doris-robot commented Mar 28, 2024

Uh oh!

doris-robot commented Mar 28, 2024

Uh oh!

mymeiyi commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

doris-robot commented Mar 29, 2024

Uh oh!

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

mymeiyi commented Apr 1, 2024

Uh oh!

mymeiyi commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

mymeiyi commented Apr 8, 2024

Uh oh!

doris-robot commented Apr 8, 2024

Uh oh!

doris-robot commented Apr 8, 2024

Uh oh!

doris-robot commented Apr 8, 2024

Uh oh!

mymeiyi commented May 7, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 7, 2024

Uh oh!

github-actionsBot commented May 7, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

mymeiyi commented May 7, 2024

Uh oh!

mymeiyi commented May 7, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

mymeiyi commented Mar 28, 2024•
edited
Loading