[feature](cloud) Add lazy commit mechanism for `commit_txn` by swjtu-zhanglei · Pull Request #38243 · apache/doris

swjtu-zhanglei · 2024-07-23T07:48:56Z

Motivation:

In cloud mode we use foundationdb for storing rowset meta, when we load data with many rowsets, commit_txn will failed because foundationdb currently limits all transactions to be below 10 MB in size, So we spilt commit_txn into several sub fdb txn to solve the problem

How:

Now commit_txn main flow like this

sub txn 1:

update partition VersionPB with txn_id
submit async task to txn_lazy_committer

txn_lazy_committer:
sub txn 2:
convert tmp rowset meta per batch
sub txn 3:
convert tmp rowset meta per batch
......
sub txn n:
make txn visible and remove txn_id in VersionPB

doris-robot · 2024-07-23T07:49:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

swjtu-zhanglei · 2024-07-23T07:49:06Z

run buildall

swjtu-zhanglei · 2024-07-23T11:15:50Z

run buildall

swjtu-zhanglei · 2024-07-23T14:13:32Z

run buildall

swjtu-zhanglei · 2024-07-25T02:02:46Z

run buildall

swjtu-zhanglei · 2024-07-25T07:06:07Z

run buildall

swjtu-zhanglei · 2024-07-25T07:27:31Z

run buildall

swjtu-zhanglei · 2024-07-25T13:24:12Z

run buildall

swjtu-zhanglei · 2024-07-25T13:28:26Z

run buildall

w41ter · 2024-07-26T09:02:26Z

+ }
+
+ err = txn->commit();
+ if (err != TxnErrorCode::TXN_OK) {


There may be conflicts caused by concurrent commits. According to our design expectations, TXN_CONFLICT also needs to be handled. One simple solution is to consider the data already converted if TXN_CONFLICT occurs (this requires that each task uses the same split batch method). Another solution is to retry once if TXN_CONFLICT occurs.
It's essential to manage the competition between transactions properly here.

swjtu-zhanglei · 2024-07-31T02:19:22Z

run buildall

swjtu-zhanglei · 2024-07-31T07:41:40Z

run buildall

github-actions

clang-tidy made some suggestions

swjtu-zhanglei · 2024-07-31T13:38:36Z

run buildall

github-actions

clang-tidy made some suggestions

doris-robot · 2024-08-27T14:13:00Z

TPC-DS: Total hot run time: 191755 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f3b5ea8bfb6214887e8d1bb42d8f87233ccc8a0, data reload: false
query1	1263	887	874	874
query2	6324	1975	1987	1975
query3	10599	3842	3974	3842
query4	59298	25088	23183	23183
query5	5434	500	509	500
query6	406	165	169	165
query7	5855	297	306	297
query8	298	212	201	201
query9	8981	2496	2446	2446
query10	486	294	269	269
query11	18335	15007	15251	15007
query12	156	101	103	101
query13	1573	418	383	383
query14	11066	6494	7551	6494
query15	256	178	183	178
query16	7517	465	483	465
query17	1132	560	555	555
query18	1897	294	303	294
query19	301	150	148	148
query20	122	109	121	109
query21	207	106	102	102
query22	4667	4361	4299	4299
query23	34328	33420	33373	33373
query24	5948	2881	2845	2845
query25	552	416	416	416
query26	688	165	167	165
query27	1785	285	282	282
query28	3702	2077	2063	2063
query29	712	425	434	425
query30	240	154	151	151
query31	917	754	763	754
query32	81	56	59	56
query33	464	292	293	292
query34	867	491	506	491
query35	846	712	720	712
query36	1083	899	969	899
query37	138	90	86	86
query38	4039	3882	3897	3882
query39	1444	1388	1413	1388
query40	190	113	116	113
query41	49	44	44	44
query42	113	96	93	93
query43	493	468	464	464
query44	1111	760	745	745
query45	200	167	166	166
query46	1108	739	747	739
query47	1861	1803	1792	1792
query48	385	303	304	303
query49	758	421	430	421
query50	828	416	433	416
query51	7102	7167	7076	7076
query52	100	86	87	86
query53	253	183	182	182
query54	588	452	461	452
query55	74	75	76	75
query56	284	262	264	262
query57	1179	1057	1096	1057
query58	227	258	230	230
query59	3140	2744	2738	2738
query60	316	278	296	278
query61	100	97	104	97
query62	754	662	672	662
query63	217	182	181	181
query64	2843	680	652	652
query65	3216	3169	3177	3169
query66	694	333	332	332
query67	15797	15109	15203	15109
query68	2989	576	575	575
query69	389	328	284	284
query70	1138	1107	1016	1016
query71	345	281	272	272
query72	6131	4019	4000	4000
query73	763	336	339	336
query74	9139	8796	8898	8796
query75	3357	2714	2727	2714
query76	1500	998	953	953
query77	527	325	317	317
query78	11320	9262	9015	9015
query79	1048	546	536	536
query80	704	518	521	518
query81	458	229	230	229
query82	226	136	133	133
query83	170	149	148	148
query84	264	78	82	78
query85	681	345	277	277
query86	304	297	307	297
query87	4380	4256	4231	4231
query88	3239	2403	2392	2392
query89	384	288	282	282
query90	1905	194	203	194
query91	122	97	102	97
query92	60	52	55	52
query93	1048	537	546	537
query94	593	301	309	301
query95	352	274	268	268
query96	598	276	272	272
query97	3158	3049	3071	3049
query98	219	206	200	200
query99	1514	1306	1285	1285
Total cold run time: 307012 ms
Total hot run time: 191755 ms

doris-robot · 2024-08-27T14:18:33Z

ClickBench: Total hot run time: 31.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f3b5ea8bfb6214887e8d1bb42d8f87233ccc8a0, data reload: false
query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.06	0.07
query5	0.52	0.48	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.49	0.48
query10	0.53	0.55	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.59
query14	0.76	0.79	0.77
query15	0.84	0.82	0.82
query16	0.39	0.37	0.37
query17	1.04	1.03	0.96
query18	0.22	0.21	0.20
query19	1.95	1.79	1.72
query20	0.02	0.00	0.01
query21	15.40	0.64	0.64
query22	4.17	7.19	2.44
query23	18.32	1.40	1.31
query24	2.06	0.22	0.22
query25	0.16	0.08	0.08
query26	0.27	0.17	0.18
query27	0.07	0.07	0.07
query28	13.24	1.01	1.00
query29	12.56	3.42	3.32
query30	0.25	0.06	0.05
query31	2.86	0.39	0.38
query32	3.28	0.48	0.47
query33	2.92	3.02	2.98
query34	16.76	4.38	4.36
query35	4.41	4.43	4.49
query36	0.66	0.49	0.48
query37	0.19	0.16	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.12 s
Total hot run time: 31.14 s

swjtu-zhanglei · 2024-08-27T14:55:16Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-08-27T15:01:16Z

+ std::unique_ptr<Transaction>& txn, MetaServiceCode& code,
+ std::string& msg);
+
+void convert_tmp_rowsets(


warning: function 'convert_tmp_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
voidconvert_tmp_rowsets( ^
Additional context
cloud/src/meta-service/txn_lazy_committer.cpp:40: 154 lines including whitespace and comments (threshold 80)
voidconvert_tmp_rowsets( ^

github-actions · 2024-08-27T15:01:17Z

+ }
+}
+
+void make_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id,


warning: function 'make_committed_txn_visible' exceeds recommended size/complexity thresholds [readability-function-size]
voidmake_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id, ^
Additional context
cloud/src/meta-service/txn_lazy_committer.cpp:200: 82 lines including whitespace and comments (threshold 80)
voidmake_committed_txn_visible(const std::string& instance_id, int64_t db_id, int64_t txn_id, ^

github-actions · 2024-08-27T15:01:17Z

+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {


warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
voidTxnLazyCommitTask::commit() { ^
Additional context
cloud/src/meta-service/txn_lazy_committer.cpp:296: 138 lines including whitespace and comments (threshold 80)
voidTxnLazyCommitTask::commit() { ^

doris-robot · 2024-08-27T15:49:30Z

TPC-H: Total hot run time: 38230 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40557e2131ead7df254d700743017f557f0777e6, data reload: false
------ Round 1 ----------------------------------
q1	17711	4543	4324	4324
q2	2018	199	171	171
q3	11601	990	1079	990
q4	10486	659	727	659
q5	7748	2839	2779	2779
q6	225	137	137	137
q7	985	617	612	612
q8	9318	2062	2084	2062
q9	7205	6534	6530	6530
q10	7012	2266	2255	2255
q11	449	246	251	246
q12	401	219	224	219
q13	17765	3017	3012	3012
q14	277	228	225	225
q15	532	483	488	483
q16	489	398	389	389
q17	974	636	726	636
q18	7507	6936	6959	6936
q19	1398	1086	1052	1052
q20	699	324	339	324
q21	4320	3156	3167	3156
q22	1120	1033	1033	1033
Total cold run time: 110240 ms
Total hot run time: 38230 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4368	4350	4321	4321
q2	382	276	269	269
q3	2895	2682	2687	2682
q4	1944	1589	1654	1589
q5	5615	5696	5751	5696
q6	231	139	142	139
q7	2246	1859	1790	1790
q8	3292	3435	3491	3435
q9	8903	8729	8782	8729
q10	3568	3422	3394	3394
q11	607	533	514	514
q12	830	674	636	636
q13	13700	3194	3203	3194
q14	323	291	291	291
q15	530	501	503	501
q16	513	448	435	435
q17	1844	1527	1533	1527
q18	8147	7748	7744	7744
q19	1752	1507	1601	1507
q20	2145	1933	1929	1929
q21	5787	5364	5580	5364
q22	1099	1059	1058	1058
Total cold run time: 70721 ms
Total hot run time: 56744 ms

doris-robot · 2024-08-27T16:01:48Z

TPC-DS: Total hot run time: 192287 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40557e2131ead7df254d700743017f557f0777e6, data reload: false
query1	1265	891	888	888
query2	6329	1912	1920	1912
query3	10584	3856	3863	3856
query4	59555	27470	23284	23284
query5	5387	499	510	499
query6	404	163	159	159
query7	5786	295	296	295
query8	280	212	216	212
query9	9076	2487	2496	2487
query10	505	267	263	263
query11	18530	14935	15357	14935
query12	155	99	101	99
query13	1568	388	381	381
query14	10750	7337	7335	7335
query15	220	172	176	172
query16	7575	477	470	470
query17	1136	586	585	585
query18	2048	312	304	304
query19	288	151	154	151
query20	119	107	111	107
query21	206	104	106	104
query22	4615	4410	4377	4377
query23	34423	33096	33491	33096
query24	6026	2825	2860	2825
query25	531	391	379	379
query26	689	155	154	154
query27	1767	284	277	277
query28	3749	2055	2045	2045
query29	691	401	402	401
query30	237	152	161	152
query31	963	755	768	755
query32	85	54	56	54
query33	468	289	275	275
query34	867	480	481	480
query35	840	698	715	698
query36	1087	922	932	922
query37	138	85	79	79
query38	3983	3863	3937	3863
query39	1455	1377	1497	1377
query40	198	116	111	111
query41	46	45	44	44
query42	119	97	96	96
query43	494	471	472	471
query44	1086	749	755	749
query45	193	168	165	165
query46	1105	759	777	759
query47	1878	1800	1806	1800
query48	376	292	295	292
query49	767	439	443	439
query50	826	427	414	414
query51	7066	7099	7031	7031
query52	101	87	88	87
query53	253	181	192	181
query54	570	477	468	468
query55	77	78	79	78
query56	302	271	274	271
query57	1188	1094	1042	1042
query58	230	234	229	229
query59	3148	2747	2942	2747
query60	310	273	290	273
query61	131	119	123	119
query62	778	667	669	667
query63	225	184	190	184
query64	2928	766	726	726
query65	3246	3167	3143	3143
query66	699	347	359	347
query67	15335	15116	15146	15116
query68	3494	586	562	562
query69	414	276	281	276
query70	1200	1088	1121	1088
query71	364	279	276	276
query72	6499	4034	3934	3934
query73	750	336	345	336
query74	9482	8793	8900	8793
query75	3364	2683	2717	2683
query76	1499	925	948	925
query77	530	327	328	327
query78	10338	9128	9029	9029
query79	1086	546	544	544
query80	874	499	524	499
query81	556	228	225	225
query82	239	135	136	135
query83	224	147	148	147
query84	261	77	75	75
query85	836	285	283	283
query86	364	293	298	293
query87	4387	4317	4202	4202
query88	3184	2386	2370	2370
query89	400	291	281	281
query90	1997	192	195	192
query91	122	100	100	100
query92	63	53	55	53
query93	1882	541	548	541
query94	778	285	306	285
query95	347	272	268	268
query96	642	278	274	274
query97	3203	3078	3075	3075
query98	220	202	203	202
query99	1577	1258	1299	1258
Total cold run time: 309231 ms
Total hot run time: 192287 ms

doris-robot · 2024-08-27T16:07:20Z

ClickBench: Total hot run time: 30.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40557e2131ead7df254d700743017f557f0777e6, data reload: false
query1	0.04	0.05	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.07	0.08
query5	0.52	0.51	0.51
query6	1.12	0.74	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.51	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.13
query13	0.62	0.60	0.59
query14	0.74	0.82	0.78
query15	0.83	0.82	0.82
query16	0.38	0.38	0.38
query17	1.07	1.06	1.04
query18	0.20	0.20	0.20
query19	1.95	1.85	1.89
query20	0.01	0.02	0.01
query21	15.41	0.65	0.65
query22	3.95	8.06	2.03
query23	18.28	1.33	1.28
query24	2.16	0.20	0.21
query25	0.16	0.08	0.07
query26	0.26	0.17	0.18
query27	0.07	0.07	0.07
query28	13.22	1.01	0.99
query29	12.63	3.37	3.37
query30	0.24	0.06	0.06
query31	2.89	0.39	0.39
query32	3.28	0.48	0.48
query33	3.01	3.02	3.01
query34	17.00	4.34	4.38
query35	4.45	4.41	4.38
query36	0.65	0.51	0.48
query37	0.20	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.14	0.14
query41	0.10	0.04	0.04
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.34 s
Total hot run time: 30.99 s

swjtu-zhanglei · 2024-08-28T03:42:32Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-08-28T03:49:51Z

+ DCHECK(txn_id > 0);
+}
+
+void TxnLazyCommitTask::commit() {


warning: function 'commit' exceeds recommended size/complexity thresholds [readability-function-size]
voidTxnLazyCommitTask::commit() { ^
Additional context
cloud/src/meta-service/txn_lazy_committer.cpp:296: 139 lines including whitespace and comments (threshold 80)
voidTxnLazyCommitTask::commit() { ^

swjtu-zhanglei · 2024-08-28T04:53:27Z

run buildall

doris-robot · 2024-08-28T05:44:53Z

TPC-H: Total hot run time: 38261 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cbbeda91329c7f9ecaf9f55ca0b98af79b86d7ee, data reload: false
------ Round 1 ----------------------------------
q1	18891	5287	4355	4355
q2	2054	189	189	189
q3	11442	964	1144	964
q4	10333	693	732	693
q5	7762	2882	2904	2882
q6	228	140	137	137
q7	974	617	601	601
q8	9327	2133	2112	2112
q9	7352	6556	6576	6556
q10	7006	2323	2297	2297
q11	441	246	245	245
q12	393	224	230	224
q13	17765	3030	3037	3030
q14	274	240	242	240
q15	528	469	488	469
q16	586	527	503	503
q17	993	694	670	670
q18	7533	6842	6862	6842
q19	1386	1016	1088	1016
q20	696	341	326	326
q21	4552	2969	2888	2888
q22	1152	1032	1022	1022
Total cold run time: 111668 ms
Total hot run time: 38261 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4435	4335	4320	4320
q2	383	271	267	267
q3	2892	2634	2744	2634
q4	1934	1718	1734	1718
q5	5725	5694	5837	5694
q6	225	133	133	133
q7	2202	1793	1847	1793
q8	3329	3488	3533	3488
q9	8904	8917	8871	8871
q10	3577	3438	3414	3414
q11	588	516	501	501
q12	865	677	678	677
q13	13058	3068	3223	3068
q14	320	278	291	278
q15	531	508	488	488
q16	648	576	602	576
q17	1851	1567	1557	1557
q18	8236	7798	7811	7798
q19	1745	1630	1537	1537
q20	2196	1923	1906	1906
q21	5786	5462	5503	5462
q22	1149	1037	1035	1035
Total cold run time: 70579 ms
Total hot run time: 57215 ms

doris-robot · 2024-08-28T05:57:17Z

TPC-DS: Total hot run time: 195283 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cbbeda91329c7f9ecaf9f55ca0b98af79b86d7ee, data reload: false
query1	1271	898	869	869
query2	6287	2025	2028	2025
query3	10783	4136	4143	4136
query4	60232	26807	23447	23447
query5	5309	500	499	499
query6	411	162	163	162
query7	5754	300	300	300
query8	316	218	218	218
query9	8643	2501	2480	2480
query10	466	269	273	269
query11	18168	15119	15538	15119
query12	160	105	102	102
query13	1502	403	403	403
query14	10627	7806	7516	7516
query15	225	182	193	182
query16	7262	500	473	473
query17	1102	593	611	593
query18	1981	311	312	311
query19	291	153	154	153
query20	122	119	123	119
query21	212	112	113	112
query22	4733	4626	4523	4523
query23	34472	33691	33805	33691
query24	6008	2936	2862	2862
query25	520	407	401	401
query26	649	165	163	163
query27	1712	279	282	279
query28	3942	2146	2142	2142
query29	678	428	429	428
query30	243	162	163	162
query31	936	771	769	769
query32	81	60	61	60
query33	428	292	293	292
query34	873	486	488	486
query35	833	722	766	722
query36	1058	958	977	958
query37	152	93	94	93
query38	3962	3815	4024	3815
query39	1460	1377	1395	1377
query40	203	123	118	118
query41	49	47	46	46
query42	114	99	101	99
query43	511	487	491	487
query44	1085	748	751	748
query45	199	167	168	167
query46	1096	773	763	763
query47	1866	1828	1802	1802
query48	385	300	304	300
query49	783	440	445	440
query50	826	440	420	420
query51	7263	7137	7115	7115
query52	97	89	90	89
query53	259	184	188	184
query54	577	478	473	473
query55	77	81	83	81
query56	279	267	270	267
query57	1192	1109	1080	1080
query58	233	250	228	228
query59	3129	2899	3049	2899
query60	296	266	275	266
query61	104	94	92	92
query62	768	664	635	635
query63	211	182	181	181
query64	2471	675	663	663
query65	3236	3161	3194	3161
query66	657	333	367	333
query67	15576	15299	15303	15299
query68	4378	571	560	560
query69	528	372	299	299
query70	1153	1067	1063	1063
query71	400	279	281	279
query72	6470	4107	4002	4002
query73	778	337	334	334
query74	9234	8848	9004	8848
query75	3389	2725	2754	2725
query76	1682	1036	1060	1036
query77	648	321	318	318
query78	11039	9489	9257	9257
query79	1976	537	547	537
query80	813	534	501	501
query81	567	236	242	236
query82	471	153	147	147
query83	175	151	150	150
query84	271	77	74	74
query85	731	292	316	292
query86	399	307	294	294
query87	4416	4276	4460	4276
query88	3373	2332	2346	2332
query89	392	285	282	282
query90	1964	193	195	193
query91	125	99	98	98
query92	58	53	52	52
query93	1505	544	548	544
query94	709	307	284	284
query95	364	271	264	264
query96	604	273	272	272
query97	3238	3105	3088	3088
query98	234	217	226	217
query99	1625	1286	1282	1282
Total cold run time: 311115 ms
Total hot run time: 195283 ms

doris-robot · 2024-08-28T06:02:57Z

ClickBench: Total hot run time: 31.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cbbeda91329c7f9ecaf9f55ca0b98af79b86d7ee, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.09
query5	0.52	0.50	0.50
query6	1.14	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.50	0.49
query10	0.55	0.53	0.54
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.61	0.59
query14	2.07	2.05	2.14
query15	0.85	0.82	0.83
query16	0.38	0.38	0.38
query17	1.08	1.07	1.01
query18	0.21	0.20	0.21
query19	1.98	1.80	1.68
query20	0.01	0.01	0.01
query21	15.42	0.68	0.66
query22	3.75	7.63	1.74
query23	18.27	1.44	1.28
query24	2.07	0.24	0.22
query25	0.14	0.09	0.08
query26	0.28	0.18	0.17
query27	0.08	0.08	0.07
query28	13.66	1.02	1.00
query29	12.67	3.41	3.34
query30	0.25	0.06	0.05
query31	2.86	0.40	0.39
query32	3.26	0.49	0.48
query33	2.97	3.06	3.02
query34	17.13	4.37	4.42
query35	4.48	4.53	4.43
query36	0.66	0.48	0.48
query37	0.19	0.16	0.15
query38	0.15	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.14
query41	0.10	0.05	0.05
query42	0.07	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 111.1 s
Total hot run time: 31.9 s

Motivation: * In cloud mode we use `foundationdb` for storing rowset meta, when we load data with many rowsets, `commit_txn` will failed because `foundationdb` currently limits all transactions to be below 10 MB in size, So we spilt `commit_txn` into several sub fdb txn to solve the problem How: `commit_txn` flow like this sub txn 1: 1. update partition `VersionPB` with txn_id 2. submit async task to txn_lazy_committer txn_lazy_committer: sub txn 2: convert tmp rowset meta per batch sub txn 3: convert tmp rowset meta per batch ...... sub txn n: make txn visible and remove txn_id in `VersionPB` `get_version` main flow like this: 1. `VersionPB` without txn_id, the same as before 2. `VersionPB` has txn_id field, it means this is a txn need be advanced by txn_lazy_committer, so submit a `txn_lazy_task` to make last txn visible and retry

swjtu-zhanglei · 2024-08-28T07:09:09Z

run buildall

doris-robot · 2024-08-28T08:08:26Z

TPC-H: Total hot run time: 38018 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f6412f7fa9a90fc50e6af06a04d4e9b42655491, data reload: false
------ Round 1 ----------------------------------
q1	18641	6512	4348	4348
q2	2027	187	175	175
q3	11731	927	1167	927
q4	10429	762	784	762
q5	7734	2851	2828	2828
q6	227	139	135	135
q7	978	612	597	597
q8	9340	2102	2094	2094
q9	7220	6535	6556	6535
q10	6999	2273	2219	2219
q11	466	249	255	249
q12	401	224	228	224
q13	18090	3009	3042	3009
q14	284	241	242	241
q15	518	481	490	481
q16	575	510	521	510
q17	991	676	713	676
q18	7504	6750	6894	6750
q19	1404	1097	979	979
q20	690	333	333	333
q21	3979	3097	2923	2923
q22	1134	1023	1040	1023
Total cold run time: 111362 ms
Total hot run time: 38018 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4371	4339	4267	4267
q2	373	274	269	269
q3	2889	2698	2647	2647
q4	1904	1722	1698	1698
q5	5698	5689	5758	5689
q6	223	131	131	131
q7	2224	1826	1851	1826
q8	3287	3449	3521	3449
q9	8854	8835	8861	8835
q10	3619	3413	3347	3347
q11	592	521	496	496
q12	847	661	670	661
q13	14093	3119	3253	3119
q14	339	289	289	289
q15	540	499	477	477
q16	625	590	576	576
q17	1871	1533	1486	1486
q18	8244	7927	7923	7923
q19	1758	1675	1565	1565
q20	2186	1876	1935	1876
q21	5852	5440	5415	5415
q22	1151	1091	1077	1077
Total cold run time: 71540 ms
Total hot run time: 57118 ms

doris-robot · 2024-08-28T08:20:37Z

TPC-DS: Total hot run time: 192851 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f6412f7fa9a90fc50e6af06a04d4e9b42655491, data reload: false
query1	1246	897	880	880
query2	6377	1979	1930	1930
query3	10770	4109	4024	4024
query4	59530	23957	23403	23403
query5	5439	497	493	493
query6	407	166	159	159
query7	5847	298	305	298
query8	292	204	220	204
query9	9045	2477	2491	2477
query10	501	272	262	262
query11	16900	15114	15206	15114
query12	159	109	106	106
query13	1623	400	396	396
query14	11171	7094	7481	7094
query15	234	171	183	171
query16	7618	447	477	447
query17	1157	551	558	551
query18	2098	307	298	298
query19	285	150	149	149
query20	122	115	110	110
query21	205	105	105	105
query22	4664	4582	4606	4582
query23	34223	33521	33123	33123
query24	5927	2826	2825	2825
query25	527	385	388	385
query26	683	150	151	150
query27	1811	272	274	272
query28	3816	2114	2100	2100
query29	690	408	407	407
query30	235	155	154	154
query31	927	725	782	725
query32	88	57	62	57
query33	471	300	291	291
query34	870	480	486	480
query35	863	725	723	723
query36	1057	902	960	902
query37	152	93	96	93
query38	3907	3845	3922	3845
query39	1457	1372	1394	1372
query40	193	114	111	111
query41	47	48	45	45
query42	112	94	93	93
query43	516	486	481	481
query44	1077	743	756	743
query45	194	163	163	163
query46	1086	743	757	743
query47	1906	1801	1833	1801
query48	359	287	300	287
query49	752	415	429	415
query50	818	418	409	409
query51	7243	7063	7089	7063
query52	95	87	84	84
query53	252	182	180	180
query54	556	446	452	446
query55	81	80	76	76
query56	281	241	241	241
query57	1208	1083	1062	1062
query58	224	222	225	222
query59	2970	2870	2743	2743
query60	305	268	260	260
query61	103	95	97	95
query62	736	664	643	643
query63	214	179	179	179
query64	2835	657	686	657
query65	3213	3161	3132	3132
query66	663	328	327	327
query67	15548	15320	15191	15191
query68	2966	569	576	569
query69	407	274	289	274
query70	1223	1089	1074	1074
query71	354	278	266	266
query72	5169	4035	3932	3932
query73	745	332	332	332
query74	9089	8856	8846	8846
query75	3351	2647	2671	2647
query76	1465	1009	1033	1009
query77	544	317	313	313
query78	9706	9374	10068	9374
query79	1088	532	538	532
query80	678	498	504	498
query81	463	237	228	228
query82	247	148	145	145
query83	172	153	148	148
query84	256	74	77	74
query85	675	271	361	271
query86	304	303	287	287
query87	4461	4199	4299	4199
query88	3296	2312	2337	2312
query89	386	282	281	281
query90	1932	194	189	189
query91	121	98	99	98
query92	60	55	52	52
query93	1049	536	542	536
query94	692	287	304	287
query95	329	258	267	258
query96	595	270	273	270
query97	3204	3041	3044	3041
query98	212	197	198	197
query99	1500	1315	1263	1263
Total cold run time: 303720 ms
Total hot run time: 192851 ms

doris-robot · 2024-08-28T08:26:15Z

ClickBench: Total hot run time: 32.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f6412f7fa9a90fc50e6af06a04d4e9b42655491, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.03	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.07
query5	0.49	0.48	0.50
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.54	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.11
query13	0.62	0.60	0.59
query14	2.02	2.07	2.10
query15	0.84	0.83	0.83
query16	0.38	0.36	0.37
query17	0.98	0.98	1.02
query18	0.20	0.20	0.20
query19	1.84	1.81	1.77
query20	0.01	0.01	0.01
query21	15.40	0.67	0.66
query22	4.64	6.18	2.61
query23	18.29	1.32	1.25
query24	2.10	0.22	0.23
query25	0.15	0.08	0.08
query26	0.28	0.18	0.17
query27	0.08	0.08	0.07
query28	13.25	1.02	0.99
query29	12.59	3.35	3.30
query30	0.24	0.06	0.05
query31	2.86	0.39	0.40
query32	3.27	0.48	0.47
query33	2.98	2.93	3.03
query34	17.29	4.35	4.41
query35	4.47	4.49	4.41
query36	0.66	0.48	0.47
query37	0.19	0.17	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.10	0.06	0.06
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 111.34 s
Total hot run time: 32.53 s

github-actions · 2024-08-29T12:25:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-29T12:25:51Z

PR approved by at least one committer and no changes requested.

github-actionsBot added the meta-change label Jul 23, 2024

This comment was marked as off-topic.
Sign in to view

swjtu-zhanglei force-pushed the master-large-txn-new branch from 0cac9c5 to d47f33aCompare July 23, 2024 11:15

swjtu-zhanglei force-pushed the master-large-txn-new branch from d47f33a to a7e2b1bCompare July 23, 2024 14:13

swjtu-zhanglei force-pushed the master-large-txn-new branch 2 times, most recently from f70c9f4 to d7c4b5cCompare July 25, 2024 07:04

swjtu-zhanglei force-pushed the master-large-txn-new branch from d7c4b5c to bf6aad6Compare July 25, 2024 07:27

This comment was marked as off-topic.
Sign in to view

swjtu-zhanglei force-pushed the master-large-txn-new branch from bf6aad6 to 9511fedCompare July 25, 2024 13:23

This comment was marked as outdated.
Sign in to view

w41ter reviewed Jul 26, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service_txn.cpp
Comment threadcloud/src/meta-service/meta_service_txn.cpp
Comment threadcloud/src/meta-service/meta_service_txn.cpp
Comment threadcloud/src/meta-service/meta_service_txn.cpp Outdated

swjtu-zhanglei force-pushed the master-large-txn-new branch from 9511fed to 8ed4422Compare July 31, 2024 02:19

swjtu-zhanglei force-pushed the master-large-txn-new branch from 8ed4422 to ae48134Compare July 31, 2024 07:41

github-actionsBot reviewed Jul 31, 2024
View reviewed changes

Comment threadcloud/src/meta-service/txn_lazy_committer.cpp

swjtu-zhanglei force-pushed the master-large-txn-new branch from ae48134 to f23de13Compare July 31, 2024 13:38

github-actionsBot reviewed Jul 31, 2024
View reviewed changes

Comment threadcloud/src/meta-service/txn_lazy_committer.cpp

swjtu-zhanglei force-pushed the master-large-txn-new branch from f23de13 to c0a6c5aCompare August 1, 2024 01:45

github-actionsBot reviewed Aug 27, 2024
View reviewed changes

gavinchou reviewed Aug 28, 2024
View reviewed changes

Comment threadcloud/src/recycler/recycler.cpp Outdated

gavinchou previously approved these changes Aug 28, 2024
View reviewed changes

github-actionsBot reviewed Aug 28, 2024
View reviewed changes

swjtu-zhanglei added 5 commits August 28, 2024 14:36

fuzzy test
8510dfb

fix some comments 20240827
7d9fae3

fuzzy test
83bd183

rename some field and function
0f6412f

gavinchou approved these changes Aug 28, 2024
View reviewed changes

w41ter approved these changes Aug 29, 2024
View reviewed changes

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

Uh oh!

Conversation

swjtu-zhanglei commented Jul 23, 2024• edited by gavinchou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

swjtu-zhanglei commented Jul 23, 2024

Uh oh!

This comment was marked as off-topic.

Uh oh!

swjtu-zhanglei commented Jul 23, 2024

Uh oh!

swjtu-zhanglei commented Jul 23, 2024

Uh oh!

swjtu-zhanglei commented Jul 25, 2024

Uh oh!

swjtu-zhanglei commented Jul 25, 2024

Uh oh!

swjtu-zhanglei commented Jul 25, 2024

Uh oh!

This comment was marked as off-topic.

Uh oh!

swjtu-zhanglei commented Jul 25, 2024

Uh oh!

swjtu-zhanglei commented Jul 25, 2024

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

w41terJul 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

swjtu-zhanglei commented Jul 31, 2024

Uh oh!

swjtu-zhanglei commented Jul 31, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

swjtu-zhanglei commented Jul 31, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

swjtu-zhanglei commented Aug 27, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotAug 27, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotAug 27, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotAug 27, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Aug 27, 2024

Uh oh!

doris-robot commented Aug 27, 2024

swjtu-zhanglei commented Jul 23, 2024•
edited by gavinchou
Loading