[fix](auto-partition) fix auto partition load lost data in multi sender by kaijchen · Pull Request #35287 · apache/doris

kaijchen · 2024-05-23T09:33:16Z

Proposed changes

Change use_cnt mechanism for incremental (auto partition) channels and streams, it's now dynamically counted.
Use close_wait() of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams.
Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table.

Replace #34740

Co-authored-by: zhaochangle zhaochangle@selectdb.com

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-23T09:33:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

kaijchen · 2024-05-23T09:33:30Z

run buildall

github-actions · 2024-05-23T09:39:18Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-23T10:50:28Z

TPC-H: Total hot run time: 41439 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8aedc7deae5d4c748225f8623b8f1adadafcfdcc, data reload: false
------ Round 1 ----------------------------------
q1	17615	4519	4349	4349
q2	2037	190	189	189
q3	10504	1263	1202	1202
q4	10192	893	818	818
q5	7486	2767	2751	2751
q6	235	133	136	133
q7	985	620	607	607
q8	9222	2141	2163	2141
q9	9738	6748	6766	6748
q10	9524	3875	3914	3875
q11	450	245	243	243
q12	456	224	239	224
q13	17181	3185	3272	3185
q14	289	243	221	221
q15	521	486	480	480
q16	494	378	398	378
q17	998	706	772	706
q18	8616	7817	7735	7735
q19	1941	1602	1569	1569
q20	641	321	319	319
q21	5278	4108	3278	3278
q22	351	288	289	288
Total cold run time: 114754 ms
Total hot run time: 41439 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4586	4422	4440	4422
q2	388	267	267	267
q3	3121	2928	2998	2928
q4	1934	1614	1626	1614
q5	5483	5570	5514	5514
q6	217	126	121	121
q7	2238	1785	1867	1785
q8	3276	3397	3440	3397
q9	8696	8784	8754	8754
q10	4032	3735	3803	3735
q11	609	507	518	507
q12	787	646	628	628
q13	16020	3201	3134	3134
q14	312	269	269	269
q15	527	491	506	491
q16	487	447	458	447
q17	1828	1511	1488	1488
q18	7805	7817	7697	7697
q19	3682	1594	1555	1555
q20	2019	1776	1776	1776
q21	9516	4782	4730	4730
q22	607	487	502	487
Total cold run time: 78170 ms
Total hot run time: 55746 ms

doris-robot · 2024-05-23T11:01:14Z

TPC-DS: Total hot run time: 169335 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8aedc7deae5d4c748225f8623b8f1adadafcfdcc, data reload: false
query1	915	384	381	381
query2	6436	2441	2322	2322
query3	6644	203	205	203
query4	19228	17477	17353	17353
query5	4153	442	415	415
query6	239	159	157	157
query7	4584	307	294	294
query8	240	186	180	180
query9	8729	2454	2455	2454
query10	454	275	322	275
query11	10680	10003	10191	10003
query12	133	92	93	92
query13	1652	365	363	363
query14	8744	6880	7910	6880
query15	203	178	180	178
query16	7197	271	264	264
query17	1672	529	513	513
query18	1821	270	271	270
query19	204	172	160	160
query20	102	87	88	87
query21	194	132	130	130
query22	4276	4097	4016	4016
query23	33682	33023	33011	33011
query24	11043	2872	2878	2872
query25	613	353	361	353
query26	1106	156	156	156
query27	2776	326	314	314
query28	7091	2089	2111	2089
query29	875	613	608	608
query30	267	168	172	168
query31	981	762	763	762
query32	90	51	55	51
query33	738	281	279	279
query34	957	489	484	484
query35	776	626	600	600
query36	1102	937	925	925
query37	125	69	75	69
query38	2866	2742	2755	2742
query39	842	786	803	786
query40	202	124	122	122
query41	45	44	42	42
query42	105	95	96	95
query43	568	543	533	533
query44	1216	729	742	729
query45	176	165	162	162
query46	1078	744	742	742
query47	1856	1765	1775	1765
query48	375	305	299	299
query49	849	371	382	371
query50	776	392	391	391
query51	6805	6732	6593	6593
query52	106	87	92	87
query53	352	284	284	284
query54	936	459	428	428
query55	72	71	73	71
query56	258	245	244	244
query57	1147	1081	1031	1031
query58	257	216	213	213
query59	3309	3141	3272	3141
query60	283	255	246	246
query61	89	89	87	87
query62	635	469	444	444
query63	306	282	286	282
query64	9800	2210	1714	1714
query65	3195	3102	3114	3102
query66	1360	328	328	328
query67	15415	14769	14968	14769
query68	4512	536	548	536
query69	449	275	275	275
query70	1183	1138	1072	1072
query71	404	270	270	270
query72	7919	2706	2575	2575
query73	705	328	327	327
query74	5988	5533	5510	5510
query75	3400	2596	2652	2596
query76	2462	952	982	952
query77	595	263	269	263
query78	10186	9757	9692	9692
query79	2306	516	526	516
query80	891	444	433	433
query81	531	247	242	242
query82	947	95	94	94
query83	263	180	168	168
query84	250	86	83	83
query85	1234	319	335	319
query86	426	292	293	292
query87	3332	3123	3086	3086
query88	3660	2425	2429	2425
query89	474	383	377	377
query90	1935	191	189	189
query91	126	97	105	97
query92	63	48	48	48
query93	1845	514	503	503
query94	1143	192	186	186
query95	409	313	308	308
query96	592	276	270	270
query97	3192	2992	3067	2992
query98	233	221	216	216
query99	1073	849	849	849
Total cold run time: 268729 ms
Total hot run time: 169335 ms

doris-robot · 2024-05-23T11:06:45Z

ClickBench: Total hot run time: 30.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8aedc7deae5d4c748225f8623b8f1adadafcfdcc, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.10	0.09
query5	0.49	0.49	0.52
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.50
query10	0.54	0.55	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.78	0.77	0.78
query15	0.83	0.82	0.81
query16	0.36	0.34	0.35
query17	1.03	0.99	1.02
query18	0.21	0.26	0.25
query19	1.90	1.77	1.79
query20	0.02	0.01	0.01
query21	15.51	0.67	0.66
query22	4.71	6.65	2.15
query23	18.29	1.38	1.23
query24	1.45	0.42	0.21
query25	0.15	0.08	0.08
query26	0.26	0.16	0.17
query27	0.08	0.08	0.08
query28	13.29	1.01	1.00
query29	13.23	3.34	3.29
query30	0.24	0.05	0.06
query31	2.86	0.38	0.37
query32	3.29	0.47	0.46
query33	2.85	2.87	2.89
query34	17.16	4.42	4.54
query35	4.49	4.50	4.57
query36	0.69	0.48	0.45
query37	0.17	0.15	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.16	0.14	0.16
query41	0.09	0.04	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.08 s
Total hot run time: 30.86 s

doris-robot · 2024-05-23T11:15:14Z

TeamCity be ut coverage result:
Function Coverage: 35.66% (9017/25287)
Line Coverage: 27.31% (74561/273034)
Region Coverage: 26.54% (38590/145406)
Branch Coverage: 23.39% (19684/84144)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8aedc7deae5d4c748225f8623b8f1adadafcfdcc_8aedc7deae5d4c748225f8623b8f1adadafcfdcc/report/index.html

kaijchen · 2024-05-24T04:15:16Z

run buildall

github-actions · 2024-05-24T04:20:27Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-24T05:27:59Z

TPC-H: Total hot run time: 41270 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a786181e6b24f8c524614630cf34328138fa552, data reload: false
------ Round 1 ----------------------------------
q1	17607	4326	4325	4325
q2	2018	180	184	180
q3	10478	1291	1204	1204
q4	10196	816	741	741
q5	7482	2725	2739	2725
q6	224	138	133	133
q7	976	611	647	611
q8	9231	2167	2099	2099
q9	9499	6723	6769	6723
q10	9547	3914	3921	3914
q11	428	241	238	238
q12	557	229	220	220
q13	17322	3200	3270	3200
q14	256	218	220	218
q15	508	472	487	472
q16	519	402	397	397
q17	981	695	743	695
q18	8467	7864	7788	7788
q19	6029	1568	1575	1568
q20	626	326	317	317
q21	5419	3241	3229	3229
q22	351	280	273	273
Total cold run time: 118721 ms
Total hot run time: 41270 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4518	4436	4410	4410
q2	377	278	273	273
q3	3158	2952	2869	2869
q4	2055	1644	1626	1626
q5	5368	5504	5505	5504
q6	219	121	124	121
q7	2227	1826	1848	1826
q8	3210	3421	3403	3403
q9	8602	8726	8654	8654
q10	4062	3719	3843	3719
q11	579	497	492	492
q12	800	631	616	616
q13	17089	3155	3123	3123
q14	302	278	294	278
q15	540	480	493	480
q16	484	485	459	459
q17	1828	1474	1493	1474
q18	7889	7547	7490	7490
q19	1762	1534	1578	1534
q20	2042	1808	1797	1797
q21	14149	4750	4925	4750
q22	590	509	488	488
Total cold run time: 81850 ms
Total hot run time: 55386 ms

doris-robot · 2024-05-24T05:38:29Z

TPC-DS: Total hot run time: 171379 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a786181e6b24f8c524614630cf34328138fa552, data reload: false
query1	936	382	376	376
query2	6434	2445	2304	2304
query3	6636	211	210	210
query4	20069	17475	17450	17450
query5	4196	425	412	412
query6	254	171	161	161
query7	4592	296	286	286
query8	234	187	172	172
query9	8702	2383	2359	2359
query10	463	286	263	263
query11	10684	10413	10308	10308
query12	138	95	89	89
query13	1656	373	369	369
query14	9266	6220	6963	6220
query15	222	165	171	165
query16	7115	260	263	260
query17	1016	518	532	518
query18	1912	266	261	261
query19	188	154	155	154
query20	91	86	89	86
query21	191	129	125	125
query22	4415	4212	4164	4164
query23	33505	33134	33014	33014
query24	5121	2827	2771	2771
query25	459	349	363	349
query26	690	152	152	152
query27	1849	327	318	318
query28	3716	2048	2042	2042
query29	852	590	589	589
query30	243	170	173	170
query31	910	760	739	739
query32	56	50	54	50
query33	389	257	254	254
query34	857	474	476	474
query35	685	595	607	595
query36	1031	902	936	902
query37	107	69	74	69
query38	2896	2781	2784	2781
query39	814	790	810	790
query40	193	120	122	120
query41	48	47	46	46
query42	106	99	98	98
query43	590	555	571	555
query44	1101	728	750	728
query45	177	195	165	165
query46	1047	712	710	710
query47	1864	1764	1800	1764
query48	370	291	302	291
query49	760	380	377	377
query50	760	400	383	383
query51	6599	6498	6696	6498
query52	99	89	89	89
query53	351	281	292	281
query54	522	451	421	421
query55	74	73	74	73
query56	260	239	242	239
query57	1085	1029	1025	1025
query58	221	205	209	205
query59	3430	3038	3032	3032
query60	267	256	247	247
query61	91	84	105	84
query62	530	460	463	460
query63	309	280	281	280
query64	2578	1796	1724	1724
query65	3160	3080	3102	3080
query66	658	334	319	319
query67	15242	14832	14913	14832
query68	4542	553	544	544
query69	420	269	264	264
query70	1189	1103	1116	1103
query71	390	268	269	268
query72	7714	5866	5267	5267
query73	709	327	322	322
query74	6045	5633	5672	5633
query75	3271	2572	2665	2572
query76	2244	1041	980	980
query77	391	261	265	261
query78	10402	9669	9752	9669
query79	2546	510	514	510
query80	1117	432	425	425
query81	533	235	241	235
query82	938	103	96	96
query83	244	172	171	171
query84	269	84	84	84
query85	984	267	257	257
query86	444	306	315	306
query87	3296	3111	3151	3111
query88	3926	2336	2321	2321
query89	464	386	378	378
query90	2029	191	189	189
query91	134	106	107	106
query92	61	54	49	49
query93	1989	523	502	502
query94	1317	194	260	194
query95	400	315	312	312
query96	591	266	268	266
query97	3155	3028	3018	3018
query98	240	232	217	217
query99	1288	852	858	852
Total cold run time: 249364 ms
Total hot run time: 171379 ms

doris-robot · 2024-05-24T05:43:59Z

ClickBench: Total hot run time: 30.47 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a786181e6b24f8c524614630cf34328138fa552, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.07
query5	0.50	0.50	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.50
query10	0.56	0.56	0.56
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.78	0.78	0.79
query15	0.82	0.82	0.81
query16	0.36	0.36	0.36
query17	0.94	1.02	1.02
query18	0.24	0.21	0.27
query19	1.75	1.72	1.73
query20	0.01	0.01	0.01
query21	15.43	0.75	0.67
query22	4.98	7.19	1.83
query23	18.26	1.38	1.30
query24	1.75	0.29	0.21
query25	0.15	0.09	0.09
query26	0.25	0.16	0.17
query27	0.08	0.07	0.08
query28	13.31	1.02	1.01
query29	13.09	3.30	3.27
query30	0.25	0.06	0.05
query31	2.85	0.38	0.37
query32	3.31	0.46	0.46
query33	2.93	2.85	2.86
query34	17.26	4.45	4.44
query35	4.50	4.40	4.64
query36	0.67	0.46	0.46
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.15
query41	0.09	0.05	0.05
query42	0.07	0.06	0.05
query43	0.04	0.03	0.03
Total cold run time: 110.4 s
Total hot run time: 30.47 s

doris-robot · 2024-05-24T05:59:42Z

TeamCity be ut coverage result:
Function Coverage: 35.64% (9018/25303)
Line Coverage: 27.30% (74581/273198)
Region Coverage: 26.53% (38604/145528)
Branch Coverage: 23.39% (19696/84192)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4a786181e6b24f8c524614630cf34328138fa552_4a786181e6b24f8c524614630cf34328138fa552/report/index.html

kaijchen · 2024-05-24T06:21:31Z

run buildall

github-actions · 2024-05-24T06:27:43Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-24T07:51:51Z

TPC-H: Total hot run time: 41401 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ccd6727eb556cccedb7e2eec79a54cf60c78a44, data reload: false
------ Round 1 ----------------------------------
q1	17615	5096	4286	4286
q2	2013	194	201	194
q3	10439	1156	1244	1156
q4	10195	791	840	791
q5	7492	2735	2731	2731
q6	233	134	137	134
q7	965	607	642	607
q8	9212	2164	2102	2102
q9	9520	6746	6803	6746
q10	9234	3884	3876	3876
q11	443	243	245	243
q12	452	237	235	235
q13	17543	3327	3132	3132
q14	250	208	228	208
q15	517	460	498	460
q16	511	387	398	387
q17	985	698	711	698
q18	8400	7897	7887	7887
q19	5606	1588	1518	1518
q20	630	333	314	314
q21	5221	4008	3408	3408
q22	362	288	289	288
Total cold run time: 117838 ms
Total hot run time: 41401 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4529	4427	4417	4417
q2	374	268	273	268
q3	3156	2977	2834	2834
q4	1897	1588	1594	1588
q5	5511	5498	5548	5498
q6	220	125	128	125
q7	2197	1830	1841	1830
q8	3234	3409	3403	3403
q9	8589	8663	8670	8663
q10	3930	3729	3871	3729
q11	611	483	504	483
q12	795	623	605	605
q13	16920	3240	3257	3240
q14	300	279	274	274
q15	525	480	484	480
q16	497	433	409	409
q17	1808	1487	1502	1487
q18	7853	7632	7484	7484
q19	1701	1544	1522	1522
q20	2050	1796	1811	1796
q21	10514	4666	4760	4666
q22	580	495	499	495
Total cold run time: 77791 ms
Total hot run time: 55296 ms

doris-robot · 2024-05-24T08:02:17Z

TPC-DS: Total hot run time: 168916 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ccd6727eb556cccedb7e2eec79a54cf60c78a44, data reload: false
query1	910	383	365	365
query2	6441	2359	2386	2359
query3	6656	202	210	202
query4	19122	17353	17145	17145
query5	4148	443	443	443
query6	242	165	165	165
query7	4594	302	295	295
query8	242	195	180	180
query9	8314	2424	2407	2407
query10	451	279	285	279
query11	10489	10133	9903	9903
query12	134	96	89	89
query13	1666	379	367	367
query14	10109	6779	6663	6663
query15	217	173	174	173
query16	7130	257	259	257
query17	1032	550	521	521
query18	1860	268	269	268
query19	201	157	156	156
query20	92	85	97	85
query21	196	126	126	126
query22	4240	3926	3887	3887
query23	33642	32955	33122	32955
query24	5089	2807	2821	2807
query25	472	370	382	370
query26	685	156	156	156
query27	1833	324	321	321
query28	3866	2087	2084	2084
query29	822	615	607	607
query30	241	171	175	171
query31	902	763	750	750
query32	58	52	54	52
query33	408	257	264	257
query34	857	477	498	477
query35	722	613	600	600
query36	1036	907	897	897
query37	109	71	75	71
query38	2881	2775	2808	2775
query39	856	789	803	789
query40	200	130	127	127
query41	44	40	44	40
query42	104	98	99	98
query43	600	557	543	543
query44	1079	742	763	742
query45	176	162	165	162
query46	1058	743	719	719
query47	1841	1772	1783	1772
query48	364	292	304	292
query49	772	383	394	383
query50	773	379	393	379
query51	6892	6701	6778	6701
query52	105	90	95	90
query53	349	313	285	285
query54	535	441	424	424
query55	74	75	73	73
query56	258	243	249	243
query57	1105	1037	1040	1037
query58	214	203	224	203
query59	3401	3277	3335	3277
query60	277	253	261	253
query61	89	89	89	89
query62	545	450	461	450
query63	315	283	285	283
query64	2660	1744	1710	1710
query65	3159	3088	3085	3085
query66	660	338	325	325
query67	15217	14709	14650	14650
query68	4540	541	546	541
query69	446	271	278	271
query70	1155	1139	1061	1061
query71	456	271	267	267
query72	7687	5454	2758	2758
query73	716	322	323	322
query74	5978	5659	5593	5593
query75	3292	2627	2631	2627
query76	2311	1051	1002	1002
query77	379	272	268	268
query78	10243	9764	9708	9708
query79	1978	521	527	521
query80	1138	445	436	436
query81	525	243	242	242
query82	932	96	96	96
query83	253	174	171	171
query84	276	85	86	85
query85	973	359	271	271
query86	458	287	289	287
query87	3317	3134	3081	3081
query88	3761	2352	2352	2352
query89	489	402	379	379
query90	2013	192	193	192
query91	126	100	99	99
query92	64	50	53	50
query93	2233	528	514	514
query94	1198	190	188	188
query95	414	317	327	317
query96	598	276	261	261
query97	3231	2994	2989	2989
query98	241	222	220	220
query99	1204	838	858	838
Total cold run time: 248456 ms
Total hot run time: 168916 ms

doris-robot · 2024-05-24T08:06:29Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (9020/25304)
Line Coverage: 27.31% (74604/273215)
Region Coverage: 26.53% (38607/145534)
Branch Coverage: 23.40% (19698/84196)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7ccd6727eb556cccedb7e2eec79a54cf60c78a44_7ccd6727eb556cccedb7e2eec79a54cf60c78a44/report/index.html

doris-robot · 2024-05-24T08:07:47Z

ClickBench: Total hot run time: 31.17 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7ccd6727eb556cccedb7e2eec79a54cf60c78a44, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.04	0.05
query3	0.22	0.05	0.06
query4	1.69	0.07	0.07
query5	0.49	0.48	0.49
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.53	0.48	0.48
query10	0.55	0.54	0.53
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	0.76	0.76	0.79
query15	0.83	0.81	0.81
query16	0.39	0.35	0.37
query17	0.95	0.94	0.96
query18	0.22	0.21	0.28
query19	1.78	1.71	1.70
query20	0.01	0.00	0.01
query21	15.75	0.66	0.66
query22	4.28	6.14	2.49
query23	18.31	1.48	1.33
query24	1.88	0.24	0.21
query25	0.15	0.09	0.08
query26	0.26	0.17	0.16
query27	0.07	0.08	0.07
query28	13.34	1.03	1.01
query29	13.86	3.38	3.37
query30	0.24	0.06	0.05
query31	2.97	0.40	0.38
query32	3.22	0.48	0.48
query33	2.88	2.89	2.92
query34	17.18	4.48	4.42
query35	4.50	4.49	4.62
query36	0.66	0.46	0.46
query37	0.19	0.15	0.17
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.06
query42	0.06	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 110.93 s
Total hot run time: 31.17 s

github-actions · 2024-05-27T03:16:02Z

PR approved by anyone and no changes requested.

…hread_load.groovy

kaijchen · 2024-05-27T03:21:45Z

run buildall

github-actions · 2024-05-27T03:27:19Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-27T04:49:46Z

TeamCity be ut coverage result:
Function Coverage: 35.73% (9009/25215)
Line Coverage: 27.36% (74562/272547)
Region Coverage: 26.57% (38566/145163)
Branch Coverage: 23.44% (19671/83928)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e8bcd2e740b49c9955c22c1c409c64af15d561e6_e8bcd2e740b49c9955c22c1c409c64af15d561e6/report/index.html

doris-robot · 2024-05-27T09:39:31Z

TPC-H: Total hot run time: 40475 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e8bcd2e740b49c9955c22c1c409c64af15d561e6, data reload: false
------ Round 1 ----------------------------------
q1	18043	4785	4395	4395
q2	2663	198	193	193
q3	11421	1178	1205	1178
q4	10864	884	804	804
q5	7572	2746	2748	2746
q6	226	136	140	136
q7	957	633	618	618
q8	9538	2109	2097	2097
q9	8976	6531	6561	6531
q10	8999	3732	3754	3732
q11	450	243	232	232
q12	436	232	225	225
q13	17768	2975	3017	2975
q14	269	229	217	217
q15	521	472	509	472
q16	525	382	374	374
q17	961	735	779	735
q18	8150	7634	7396	7396
q19	7243	1582	1538	1538
q20	654	310	300	300
q21	4890	3303	3701	3303
q22	346	278	283	278
Total cold run time: 121472 ms
Total hot run time: 40475 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4371	4250	4249	4249
q2	367	255	268	255
q3	2968	2817	2787	2787
q4	1851	1581	1519	1519
q5	5231	5261	5318	5261
q6	222	125	128	125
q7	2119	1680	1721	1680
q8	3206	3379	3330	3330
q9	8352	8276	8313	8276
q10	3901	3678	3701	3678
q11	599	493	486	486
q12	767	587	595	587
q13	17406	2964	2970	2964
q14	298	254	263	254
q15	517	477	479	477
q16	473	418	405	405
q17	1769	1511	1467	1467
q18	7693	7605	7404	7404
q19	1660	1567	1566	1566
q20	1969	1804	1797	1797
q21	4949	4692	4727	4692
q22	567	480	491	480
Total cold run time: 71255 ms
Total hot run time: 53739 ms

doris-robot · 2024-05-27T09:50:35Z

TPC-DS: Total hot run time: 172477 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e8bcd2e740b49c9955c22c1c409c64af15d561e6, data reload: false
query1	916	380	371	371
query2	6473	2425	2293	2293
query3	6647	205	210	205
query4	22588	17437	17185	17185
query5	4159	441	464	441
query6	244	160	156	156
query7	4589	313	301	301
query8	240	195	184	184
query9	8467	2359	2353	2353
query10	446	280	265	265
query11	10497	10108	9904	9904
query12	145	88	88	88
query13	1639	372	390	372
query14	8505	7932	7128	7128
query15	258	170	171	170
query16	8163	279	263	263
query17	1831	548	543	543
query18	2145	280	271	271
query19	209	159	160	159
query20	91	89	84	84
query21	193	139	138	138
query22	4104	3944	3901	3901
query23	33756	33278	33363	33278
query24	11671	2901	2824	2824
query25	662	353	362	353
query26	1768	157	157	157
query27	2954	333	332	332
query28	7616	2048	2032	2032
query29	1047	615	604	604
query30	290	150	150	150
query31	956	753	736	736
query32	96	55	55	55
query33	758	269	264	264
query34	1021	491	505	491
query35	747	626	615	615
query36	1068	912	924	912
query37	275	68	67	67
query38	2883	2753	2778	2753
query39	829	800	823	800
query40	268	130	125	125
query41	45	42	43	42
query42	107	97	99	97
query43	599	567	582	567
query44	1209	744	766	744
query45	174	160	164	160
query46	1071	709	745	709
query47	1836	1764	1769	1764
query48	374	314	298	298
query49	1186	403	408	403
query50	779	399	412	399
query51	6850	6750	6651	6651
query52	115	94	91	91
query53	355	307	284	284
query54	972	449	422	422
query55	78	73	71	71
query56	284	244	244	244
query57	1126	1032	1088	1032
query58	232	203	216	203
query59	3456	3097	3321	3097
query60	285	256	262	256
query61	89	85	87	85
query62	650	480	471	471
query63	327	295	298	295
query64	9787	2235	1797	1797
query65	3184	3124	3106	3106
query66	1389	325	328	325
query67	15257	14715	14674	14674
query68	5249	535	540	535
query69	490	287	275	275
query70	1097	1139	1168	1139
query71	466	279	268	268
query72	7269	5547	5459	5459
query73	758	330	324	324
query74	6171	5576	5652	5576
query75	3860	2653	2649	2649
query76	3713	981	1040	981
query77	615	296	274	274
query78	10360	9775	9665	9665
query79	2262	516	510	510
query80	1091	447	478	447
query81	503	218	220	218
query82	690	101	92	92
query83	203	173	174	173
query84	265	87	94	87
query85	1471	281	270	270
query86	456	277	315	277
query87	3307	3130	3098	3098
query88	4076	2453	2535	2453
query89	503	397	387	387
query90	2013	192	192	192
query91	123	99	97	97
query92	65	47	49	47
query93	1951	528	499	499
query94	1126	192	186	186
query95	405	321	320	320
query96	600	283	278	278
query97	3217	3042	3063	3042
query98	240	217	214	214
query99	1249	859	835	835
Total cold run time: 278894 ms
Total hot run time: 172477 ms

doris-robot · 2024-05-27T09:56:04Z

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e8bcd2e740b49c9955c22c1c409c64af15d561e6, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.48	0.50
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.56	0.53
query11	0.16	0.12	0.11
query12	0.15	0.11	0.11
query13	0.59	0.59	0.59
query14	0.75	0.75	0.78
query15	0.82	0.80	0.80
query16	0.35	0.35	0.36
query17	0.93	0.94	0.97
query18	0.21	0.25	0.25
query19	1.76	1.67	1.68
query20	0.02	0.01	0.01
query21	15.60	0.69	0.67
query22	4.25	7.14	2.08
query23	18.27	1.37	1.28
query24	1.29	0.42	0.21
query25	0.14	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.08	0.08
query28	13.41	1.01	1.00
query29	12.67	3.29	3.29
query30	0.26	0.07	0.06
query31	2.86	0.38	0.39
query32	3.28	0.46	0.47
query33	2.85	2.94	2.90
query34	17.14	4.40	4.39
query35	4.49	4.45	4.49
query36	0.64	0.46	0.45
query37	0.18	0.15	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.79 s
Total hot run time: 30.56 s

zclllyybb

LGTM

liaoxin01

LGTM

github-actions · 2024-05-29T06:19:23Z

PR approved by at least one committer and no changes requested.

…er (apache#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace apache#34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>

…er (#35287) (#35630) ## Proposed changes Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Backport #35287 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>

…er (#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace #34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>

…lti sender (#35287)" (#36098) Reverts #35630 because it brought some more damaging bugs. we will fix it and merge in next version

…er (apache#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace apache#34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>

…er (apache#35287) (apache#35630) ## Proposed changes Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Backport apache#35287 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>

…er (apache#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace apache#34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>

zclllyybband others added 7 commits May 23, 2024 17:04

1
77b766d

make hang on reciever
a43fae6

just use mark_close
1b775f0

testcases
a28d007

temporarily change some cases
cea3de7

make non-auto table not hang on close. add comment.
05bc2bf

test
8aedc7d

github-actionsBot added the meta-change label May 23, 2024

morrySnow marked this pull request as draft May 23, 2024 09:38

fake tablet id always 0
7ccd672

kaijchen force-pushed the ap_fix_load branch from 4a78618 to 7ccd672Compare May 24, 2024 06:21

github-actionsBot added the reviewed label May 27, 2024

kaijchen commented May 27, 2024
View reviewed changes

Comment threadregression-test/suites/partition_p1/auto_partition/sql/multi_thread_load.groovy Outdated

revert regression-test/suites/partition_p1/auto_partition/sql/multi_t…
e8bcd2e
…hread_load.groovy

zclllyybb approved these changes May 28, 2024
View reviewed changes

liaoxin01 approved these changes May 29, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 29, 2024

liaoxin01 merged commit e2d7eca into apache:masterMay 29, 2024

yiguolei added the 2.1.0-conflict label May 29, 2024

kaijchen mentioned this pull request May 30, 2024
[fix](auto-partition) fix auto partition load lost data in multi sender (#35287) #35630
Merged

dataroaring added dev/2.1.4-merged and removed 2.1.0-conflict labels Jun 11, 2024

morningman pushed a commit that referenced this pull request Jun 11, 2024
Revert "[fix](auto-partition) fix auto partition load lost data in mu…
3b23eee
…lti sender (#35287)" (#36098) Reverts #35630 because it brought some more damaging bugs. we will fix it and merge in next version

kaijchen mentioned this pull request Jun 19, 2024
[fix](auto-partition) fix auto partition load lost data in multi sender #36550
Closed

w41ter mentioned this pull request Jul 8, 2024
[fix](fe) Fix the sql of AddPartitionRecord #37339
Closed

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Uh oh!

Conversation

kaijchen commented May 23, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

kaijchen commented May 23, 2024

Uh oh!

github-actionsBot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

kaijchen commented May 24, 2024

Uh oh!

github-actionsBot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

kaijchen commented May 24, 2024

Uh oh!

github-actionsBot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

doris-robot commented May 24, 2024

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

Uh oh!

kaijchen commented May 27, 2024

Uh oh!

github-actionsBot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

doris-robot commented May 27, 2024

Uh oh!

zclllyybb left a comment

Choose a reason for hiding this comment

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kaijchen commented May 23, 2024•
edited
Loading