[Refactor](shuffle) Fix wrong deal of batching block in TabletSinkHashPartitioner by zclllyybb · Pull Request #48720 · apache/doris

zclllyybb · 2025-03-05T20:55:32Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: bug introduced by #44102, expanded by #45256, partially fixed by #47911

Problem Summary:

also fixed Auto Partition + Auto Increment-Column may core dump in loading

Release note

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-03-05T20:55:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zclllyybb · 2025-03-05T20:56:34Z

run buildall

zclllyybb · 2025-03-05T21:08:02Z

run buildall

doris-robot · 2025-03-05T21:44:39Z

TPC-H: Total hot run time: 32869 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a86da5e6eadb4fbac8772fdb5bfdaaf147f6000, data reload: false
------ Round 1 ----------------------------------
q1	17605	5246	5148	5148
q2	2059	301	177	177
q3	10419	1258	773	773
q4	10253	1026	550	550
q5	7997	2355	2364	2355
q6	188	169	140	140
q7	923	753	602	602
q8	9299	1269	1100	1100
q9	4984	4943	4768	4768
q10	6823	2316	1909	1909
q11	483	285	257	257
q12	343	351	225	225
q13	17784	3703	3114	3114
q14	225	225	208	208
q15	534	493	489	489
q16	618	630	603	603
q17	608	848	386	386
q18	6764	6441	6354	6354
q19	1884	952	559	559
q20	310	329	188	188
q21	2857	2139	1973	1973
q22	1092	1041	991	991
Total cold run time: 104052 ms
Total hot run time: 32869 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5290	5199	5200	5199
q2	241	327	240	240
q3	2188	2699	2358	2358
q4	1410	1847	1344	1344
q5	4240	4116	4178	4116
q6	207	161	123	123
q7	1942	1936	1809	1809
q8	2627	2700	2521	2521
q9	7256	7261	7348	7261
q10	3067	3215	2804	2804
q11	603	524	489	489
q12	655	722	591	591
q13	3528	3924	3361	3361
q14	289	315	279	279
q15	508	491	463	463
q16	641	697	666	666
q17	1158	1580	1347	1347
q18	7755	7635	7552	7552
q19	904	888	987	888
q20	1975	2045	1879	1879
q21	5496	4932	4828	4828
q22	1080	1120	1020	1020
Total cold run time: 53060 ms
Total hot run time: 51138 ms

doris-robot · 2025-03-05T21:56:55Z

TPC-DS: Total hot run time: 192644 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a86da5e6eadb4fbac8772fdb5bfdaaf147f6000, data reload: false
query1	1411	1059	1026	1026
query2	6223	2000	1978	1978
query3	10986	4727	4604	4604
query4	53722	24474	23004	23004
query5	5202	525	511	511
query6	388	213	208	208
query7	5127	511	291	291
query8	346	249	240	240
query9	6774	2579	2573	2573
query10	430	339	259	259
query11	15372	15091	14849	14849
query12	170	112	107	107
query13	1174	513	415	415
query14	10571	6846	6515	6515
query15	217	198	179	179
query16	6976	617	474	474
query17	1096	718	570	570
query18	1537	410	335	335
query19	199	200	167	167
query20	130	128	169	128
query21	212	128	113	113
query22	4323	4385	4265	4265
query23	34019	33905	33596	33596
query24	6040	2410	2431	2410
query25	467	487	400	400
query26	764	279	155	155
query27	2018	510	337	337
query28	2815	2511	2453	2453
query29	586	572	430	430
query30	274	220	210	210
query31	893	843	811	811
query32	69	63	60	60
query33	457	359	303	303
query34	796	859	517	517
query35	815	851	782	782
query36	968	1007	905	905
query37	122	103	75	75
query38	4278	4405	4304	4304
query39	1491	1443	1416	1416
query40	215	117	103	103
query41	55	62	62	62
query42	128	115	117	115
query43	535	540	513	513
query44	1395	830	812	812
query45	193	177	172	172
query46	854	1028	652	652
query47	1820	1829	1812	1812
query48	387	411	316	316
query49	720	510	453	453
query50	708	740	444	444
query51	4351	4289	4282	4282
query52	100	106	97	97
query53	238	250	183	183
query54	496	518	451	451
query55	81	84	78	78
query56	303	264	250	250
query57	1173	1194	1141	1141
query58	248	243	247	243
query59	2696	2937	2842	2842
query60	296	296	258	258
query61	129	121	118	118
query62	749	736	698	698
query63	237	197	197	197
query64	2105	1042	719	719
query65	4537	4512	4333	4333
query66	758	391	339	339
query67	15811	15499	15253	15253
query68	7140	888	516	516
query69	540	297	267	267
query70	1235	1134	1137	1134
query71	504	289	277	277
query72	5797	3782	3816	3782
query73	1223	761	341	341
query74	9251	9033	8839	8839
query75	3661	3144	2721	2721
query76	4253	1183	774	774
query77	620	367	273	273
query78	10126	10086	9329	9329
query79	3770	826	564	564
query80	713	522	444	444
query81	495	276	227	227
query82	707	125	93	93
query83	278	166	149	149
query84	288	90	76	76
query85	792	370	310	310
query86	408	292	296	292
query87	4545	4444	4343	4343
query88	3405	2205	2192	2192
query89	430	330	288	288
query90	1936	195	192	192
query91	134	134	109	109
query92	77	61	54	54
query93	2657	1020	569	569
query94	651	414	293	293
query95	353	269	266	266
query96	489	563	266	266
query97	3338	3436	3287	3287
query98	248	200	195	195
query99	1450	1407	1271	1271
Total cold run time: 302290 ms
Total hot run time: 192644 ms

doris-robot · 2025-03-05T22:02:13Z

ClickBench: Total hot run time: 31.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a86da5e6eadb4fbac8772fdb5bfdaaf147f6000, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.56	0.54	0.55
query6	1.18	0.72	0.72
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.53
query10	0.58	0.57	0.59
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.59
query14	2.72	2.71	2.69
query15	0.93	0.87	0.86
query16	0.40	0.38	0.37
query17	1.03	1.03	1.01
query18	0.21	0.19	0.19
query19	1.89	1.79	1.97
query20	0.01	0.01	0.02
query21	15.35	0.92	0.57
query22	0.76	1.10	0.59
query23	15.10	1.36	0.62
query24	7.25	1.12	1.61
query25	0.52	0.30	0.14
query26	0.61	0.16	0.14
query27	0.05	0.05	0.05
query28	10.02	0.79	0.42
query29	12.58	3.91	3.32
query30	0.26	0.09	0.07
query31	3.09	0.58	0.39
query32	3.23	0.55	0.46
query33	2.98	2.97	2.96
query34	15.74	5.10	4.53
query35	4.50	4.54	4.52
query36	0.68	0.50	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.29 s
Total hot run time: 31.03 s

doris-robot · 2025-03-05T22:47:05Z

BE UT Coverage Report

Increment line coverage 3.00% (3/100) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.93% (12278/26732)
Line Coverage	35.39% (103659/292945)
Region Coverage	34.55% (53097/153701)
Branch Coverage	30.23% (26882/88914)

yiguolei

These files should add BE UT

zclllyybb · 2025-03-06T15:18:05Z

run buildall

doris-robot · 2025-03-06T15:56:03Z

TPC-H: Total hot run time: 32475 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd24ddd638a82398053c2992892186a232d92554, data reload: false
------ Round 1 ----------------------------------
q1	17587	5135	5028	5028
q2	2041	292	183	183
q3	10438	1288	769	769
q4	10262	1020	525	525
q5	7955	2429	2342	2342
q6	197	166	131	131
q7	921	747	594	594
q8	9298	1275	1087	1087
q9	4997	4639	4769	4639
q10	6818	2338	1899	1899
q11	461	270	263	263
q12	349	354	219	219
q13	17766	3697	3051	3051
q14	236	217	203	203
q15	542	506	498	498
q16	654	615	604	604
q17	562	868	350	350
q18	6759	6561	6373	6373
q19	1915	941	543	543
q20	308	321	190	190
q21	2786	2177	1953	1953
q22	1097	1043	1031	1031
Total cold run time: 103949 ms
Total hot run time: 32475 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5179	5118	5097	5097
q2	232	332	233	233
q3	2140	2677	2247	2247
q4	1440	1904	1391	1391
q5	4270	4082	4185	4082
q6	208	165	122	122
q7	1884	1887	1737	1737
q8	2650	2704	2583	2583
q9	7283	7220	7156	7156
q10	3033	3254	2774	2774
q11	581	513	491	491
q12	728	724	568	568
q13	3457	3968	3269	3269
q14	274	302	282	282
q15	520	492	478	478
q16	648	685	623	623
q17	1146	1629	1312	1312
q18	7875	7726	7574	7574
q19	825	781	884	781
q20	1965	2038	1900	1900
q21	5494	4922	4675	4675
q22	1112	1076	1083	1076
Total cold run time: 52944 ms
Total hot run time: 50451 ms

doris-robot · 2025-03-06T16:08:21Z

TPC-DS: Total hot run time: 192865 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd24ddd638a82398053c2992892186a232d92554, data reload: false
query1	1359	996	1025	996
query2	6091	1966	1954	1954
query3	11162	4613	4608	4608
query4	56298	25563	23393	23393
query5	4950	548	493	493
query6	339	207	200	200
query7	4891	502	289	289
query8	335	243	236	236
query9	5741	2602	2602	2602
query10	431	306	272	272
query11	15308	15106	14977	14977
query12	149	107	110	107
query13	1079	531	396	396
query14	10571	6772	6305	6305
query15	193	198	175	175
query16	7062	671	492	492
query17	1060	698	574	574
query18	1521	406	318	318
query19	204	200	166	166
query20	128	131	124	124
query21	214	128	106	106
query22	4584	4516	4274	4274
query23	34022	33512	33450	33450
query24	5813	2391	2420	2391
query25	461	465	405	405
query26	666	280	160	160
query27	1665	497	331	331
query28	2787	2495	2478	2478
query29	591	552	442	442
query30	273	219	198	198
query31	867	875	806	806
query32	68	65	63	63
query33	436	357	343	343
query34	762	872	514	514
query35	831	852	769	769
query36	931	1009	921	921
query37	118	97	77	77
query38	4144	4195	4310	4195
query39	1460	1463	1430	1430
query40	221	121	110	110
query41	63	56	62	56
query42	122	108	104	104
query43	502	538	497	497
query44	1331	837	826	826
query45	193	179	167	167
query46	873	1044	669	669
query47	1860	1919	1814	1814
query48	413	423	316	316
query49	697	532	429	429
query50	712	756	438	438
query51	4286	4286	4288	4286
query52	115	102	90	90
query53	234	278	193	193
query54	501	514	421	421
query55	125	83	81	81
query56	284	275	236	236
query57	1172	1184	1132	1132
query58	248	240	228	228
query59	2769	2887	2871	2871
query60	287	281	261	261
query61	119	123	124	123
query62	740	743	688	688
query63	250	189	194	189
query64	1456	1057	700	700
query65	4526	4467	4416	4416
query66	758	401	347	347
query67	16005	15777	15367	15367
query68	6940	877	512	512
query69	538	305	263	263
query70	1182	1132	1086	1086
query71	506	285	272	272
query72	5814	3767	3780	3767
query73	1363	760	346	346
query74	9176	8902	8827	8827
query75	3823	3161	2702	2702
query76	4170	1204	760	760
query77	641	373	282	282
query78	10122	9995	9293	9293
query79	4460	823	572	572
query80	728	576	458	458
query81	498	260	225	225
query82	692	130	93	93
query83	293	172	147	147
query84	286	92	75	75
query85	788	349	321	321
query86	392	309	309	309
query87	4536	4452	4419	4419
query88	3540	2222	2250	2222
query89	451	308	282	282
query90	1821	213	214	213
query91	139	135	109	109
query92	80	57	58	57
query93	3094	1031	567	567
query94	661	425	302	302
query95	400	270	266	266
query96	487	569	288	288
query97	3345	3425	3284	3284
query98	231	204	204	204
query99	1432	1403	1257	1257
Total cold run time: 303056 ms
Total hot run time: 192865 ms

doris-robot · 2025-03-06T16:13:38Z

ClickBench: Total hot run time: 30.96 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd24ddd638a82398053c2992892186a232d92554, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.10
query5	0.57	0.56	0.54
query6	1.18	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.53
query10	0.58	0.60	0.58
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.60
query14	2.74	2.71	2.85
query15	0.92	0.85	0.86
query16	0.38	0.39	0.40
query17	1.04	1.06	1.01
query18	0.22	0.19	0.20
query19	1.90	1.88	2.00
query20	0.02	0.01	0.02
query21	15.35	0.90	0.53
query22	0.76	1.19	0.78
query23	14.76	1.35	0.63
query24	6.62	1.45	0.73
query25	0.54	0.28	0.07
query26	0.49	0.17	0.15
query27	0.06	0.06	0.05
query28	9.24	0.89	0.44
query29	12.53	4.03	3.36
query30	0.25	0.09	0.06
query31	2.81	0.59	0.39
query32	3.23	0.56	0.46
query33	3.02	3.00	3.06
query34	15.82	5.17	4.50
query35	4.59	4.58	4.52
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.29 s
Total hot run time: 30.96 s

zclllyybb · 2025-05-14T07:40:29Z

run buildall

doris-robot · 2025-05-14T08:08:29Z

TPC-H: Total hot run time: 33882 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e74e9c5bd79cf89dff7711040c93f57cf3789812, data reload: false
------ Round 1 ----------------------------------
q1	26181	5022	4947	4947
q2	2062	270	182	182
q3	10484	1241	724	724
q4	10227	1016	562	562
q5	7753	2410	2345	2345
q6	189	165	133	133
q7	925	782	618	618
q8	9322	1275	1047	1047
q9	6791	5073	5079	5073
q10	6848	2327	1886	1886
q11	481	283	264	264
q12	342	345	213	213
q13	17777	3691	3097	3097
q14	238	230	219	219
q15	537	488	495	488
q16	416	426	380	380
q17	600	860	368	368
q18	7632	7322	7071	7071
q19	1476	947	567	567
q20	335	354	222	222
q21	4037	3412	2488	2488
q22	1070	1023	988	988
Total cold run time: 115723 ms
Total hot run time: 33882 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5123	5081	5043	5043
q2	233	350	234	234
q3	2166	2650	2314	2314
q4	1435	1839	1386	1386
q5	4567	4443	4425	4425
q6	212	167	128	128
q7	1965	1887	1750	1750
q8	2592	2523	2468	2468
q9	7175	7030	7034	7030
q10	3024	3156	2756	2756
q11	577	523	514	514
q12	688	755	590	590
q13	3535	3865	3318	3318
q14	283	292	268	268
q15	516	481	462	462
q16	452	483	430	430
q17	1138	1575	1375	1375
q18	7797	7462	7337	7337
q19	791	773	853	773
q20	1989	2084	1965	1965
q21	5126	4708	4576	4576
q22	1074	1048	1011	1011
Total cold run time: 52458 ms
Total hot run time: 50153 ms

doris-robot · 2025-05-14T08:20:07Z

TPC-DS: Total hot run time: 187015 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e74e9c5bd79cf89dff7711040c93f57cf3789812, data reload: false
query1	1017	473	477	473
query2	6560	1908	1848	1848
query3	6769	221	221	221
query4	26342	24121	23530	23530
query5	4655	618	458	458
query6	288	196	222	196
query7	4625	503	289	289
query8	283	240	224	224
query9	8636	2655	2654	2654
query10	513	330	273	273
query11	15357	15076	14845	14845
query12	155	107	105	105
query13	1653	530	410	410
query14	8728	6044	6154	6044
query15	216	188	171	171
query16	7134	644	479	479
query17	1082	729	584	584
query18	1976	403	308	308
query19	197	188	176	176
query20	119	119	120	119
query21	214	122	108	108
query22	4114	4087	4084	4084
query23	34446	33170	32987	32987
query24	8109	2467	2385	2385
query25	546	443	395	395
query26	1231	267	156	156
query27	2744	493	336	336
query28	4345	2133	2103	2103
query29	759	553	431	431
query30	290	214	193	193
query31	978	861	759	759
query32	73	65	63	63
query33	556	375	303	303
query34	799	861	506	506
query35	778	799	732	732
query36	924	982	885	885
query37	111	104	75	75
query38	4164	4227	4231	4227
query39	1468	1391	1406	1391
query40	214	118	106	106
query41	57	56	53	53
query42	123	110	106	106
query43	505	503	460	460
query44	1281	807	794	794
query45	175	174	163	163
query46	844	1022	624	624
query47	1777	1814	1735	1735
query48	380	409	314	314
query49	788	523	417	417
query50	644	679	394	394
query51	4152	4107	4146	4107
query52	109	109	96	96
query53	219	256	194	194
query54	615	567	504	504
query55	81	79	84	79
query56	293	300	296	296
query57	1160	1148	1122	1122
query58	260	266	249	249
query59	2623	2644	2643	2643
query60	315	305	315	305
query61	120	124	118	118
query62	788	741	664	664
query63	226	187	186	186
query64	4383	1010	653	653
query65	4355	4232	4318	4232
query66	1170	448	333	333
query67	15894	15424	15815	15424
query68	8058	889	512	512
query69	496	292	325	292
query70	1277	1143	1106	1106
query71	470	320	294	294
query72	5778	4773	4812	4773
query73	717	617	342	342
query74	9226	9168	9007	9007
query75	3783	3223	2737	2737
query76	3622	1192	743	743
query77	791	380	287	287
query78	10157	9982	9369	9369
query79	2725	827	572	572
query80	621	536	448	448
query81	481	267	224	224
query82	550	131	98	98
query83	291	248	241	241
query84	298	109	83	83
query85	805	355	316	316
query86	387	298	305	298
query87	4541	4456	4433	4433
query88	3593	2310	2241	2241
query89	378	329	277	277
query90	1814	201	261	201
query91	135	159	113	113
query92	76	66	56	56
query93	1820	947	572	572
query94	669	418	304	304
query95	377	292	277	277
query96	503	553	277	277
query97	3182	3217	3119	3119
query98	233	209	206	206
query99	1438	1398	1300	1300
Total cold run time: 275797 ms
Total hot run time: 187015 ms

doris-robot · 2025-05-14T08:25:17Z

ClickBench: Total hot run time: 29.84 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e74e9c5bd79cf89dff7711040c93f57cf3789812, data reload: false
query1	0.04	0.03	0.03
query2	0.12	0.10	0.10
query3	0.26	0.20	0.19
query4	1.59	0.20	0.19
query5	0.60	0.57	0.60
query6	1.19	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.57	0.54	0.52
query10	0.58	0.58	0.57
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	0.79	0.82	0.80
query15	0.87	0.85	0.84
query16	0.38	0.38	0.38
query17	1.07	1.06	1.02
query18	0.23	0.21	0.21
query19	1.88	1.84	1.90
query20	0.01	0.01	0.01
query21	15.42	0.91	0.57
query22	0.75	1.16	0.61
query23	15.06	1.37	0.61
query24	6.58	1.37	1.24
query25	0.46	0.16	0.26
query26	0.66	0.16	0.13
query27	0.05	0.05	0.05
query28	9.40	0.90	0.44
query29	12.54	3.98	3.41
query30	0.25	0.11	0.06
query31	2.82	0.60	0.39
query32	3.23	0.55	0.46
query33	3.11	3.06	3.14
query34	15.94	5.16	4.52
query35	4.53	4.53	4.50
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.09 s
Total hot run time: 29.84 s

zclllyybb · 2025-05-14T17:13:30Z

run buildall

doris-robot · 2025-05-14T18:00:59Z

TPC-H: Total hot run time: 33901 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c6f154d765942727ba133703243016ae76a32ce, data reload: false
------ Round 1 ----------------------------------
q1	26061	5086	5062	5062
q2	2074	277	192	192
q3	10394	1245	694	694
q4	10223	1016	535	535
q5	7541	2405	2361	2361
q6	191	170	138	138
q7	904	731	620	620
q8	9337	1298	1103	1103
q9	6800	5199	5055	5055
q10	6877	2331	1897	1897
q11	490	289	273	273
q12	351	349	238	238
q13	17793	3702	3164	3164
q14	231	226	209	209
q15	528	493	512	493
q16	427	432	369	369
q17	604	868	361	361
q18	7536	7184	7071	7071
q19	1621	966	558	558
q20	345	346	212	212
q21	4191	2616	2327	2327
q22	1054	989	969	969
Total cold run time: 115573 ms
Total hot run time: 33901 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5264	5132	5133	5132
q2	237	316	224	224
q3	2179	2674	2310	2310
q4	1363	1795	1339	1339
q5	4429	4375	4431	4375
q6	212	170	127	127
q7	2001	1962	1778	1778
q8	2619	2678	2541	2541
q9	7130	7139	6962	6962
q10	3073	3174	2786	2786
q11	598	532	495	495
q12	703	791	641	641
q13	3478	3918	3288	3288
q14	299	292	273	273
q15	558	531	505	505
q16	440	472	455	455
q17	1173	1582	1414	1414
q18	7723	7535	7430	7430
q19	862	875	892	875
q20	1968	2051	1854	1854
q21	4859	4478	4512	4478
q22	1102	1069	1014	1014
Total cold run time: 52270 ms
Total hot run time: 50296 ms

doris-robot · 2025-05-14T18:13:15Z

TPC-DS: Total hot run time: 193904 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c6f154d765942727ba133703243016ae76a32ce, data reload: false
query1	1428	1099	1054	1054
query2	6224	1796	1754	1754
query3	11003	4532	4538	4532
query4	54859	25840	23573	23573
query5	5003	509	457	457
query6	345	230	193	193
query7	4867	520	302	302
query8	311	248	244	244
query9	5335	2681	2676	2676
query10	451	340	275	275
query11	15047	15091	14769	14769
query12	168	113	103	103
query13	1033	543	408	408
query14	10233	6400	6434	6400
query15	206	213	185	185
query16	7118	656	522	522
query17	1101	760	617	617
query18	1623	413	305	305
query19	201	199	162	162
query20	122	116	123	116
query21	208	123	109	109
query22	4404	4600	4303	4303
query23	34359	33689	33805	33689
query24	6558	2469	2471	2469
query25	463	482	414	414
query26	717	277	148	148
query27	2303	517	339	339
query28	3075	2199	2168	2168
query29	584	587	473	473
query30	278	226	192	192
query31	805	861	777	777
query32	79	69	60	60
query33	459	358	306	306
query34	792	860	547	547
query35	794	875	738	738
query36	951	995	890	890
query37	115	101	73	73
query38	4253	4261	4196	4196
query39	1480	1483	1428	1428
query40	208	125	108	108
query41	57	55	52	52
query42	126	112	107	107
query43	504	520	466	466
query44	1349	860	829	829
query45	179	177	167	167
query46	856	1052	640	640
query47	1871	1878	1812	1812
query48	415	434	334	334
query49	696	544	449	449
query50	673	707	403	403
query51	4299	4272	4264	4264
query52	114	114	109	109
query53	231	271	197	197
query54	598	615	538	538
query55	86	80	88	80
query56	326	315	275	275
query57	1213	1217	1124	1124
query58	253	258	256	256
query59	2640	2766	2728	2728
query60	342	356	325	325
query61	123	121	121	121
query62	737	721	701	701
query63	240	192	197	192
query64	1942	1042	737	737
query65	4305	4276	4221	4221
query66	721	406	306	306
query67	15975	15784	15495	15495
query68	7179	913	520	520
query69	541	311	272	272
query70	1197	1139	1133	1133
query71	506	334	296	296
query72	5862	4676	4709	4676
query73	1512	569	348	348
query74	8982	9034	8809	8809
query75	3880	3244	2762	2762
query76	4132	1190	747	747
query77	641	373	291	291
query78	10198	10122	9352	9352
query79	2395	824	588	588
query80	641	505	442	442
query81	487	249	221	221
query82	269	135	94	94
query83	249	245	228	228
query84	310	104	89	89
query85	814	364	308	308
query86	347	297	308	297
query87	4473	4492	4331	4331
query88	3739	2319	2322	2319
query89	408	320	291	291
query90	1763	215	221	215
query91	145	146	112	112
query92	75	64	59	59
query93	1939	954	585	585
query94	614	414	306	306
query95	371	293	286	286
query96	520	568	293	293
query97	3166	3176	3167	3167
query98	229	215	210	210
query99	1350	1408	1255	1255
Total cold run time: 299368 ms
Total hot run time: 193904 ms

doris-robot · 2025-05-14T18:18:28Z

ClickBench: Total hot run time: 29.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c6f154d765942727ba133703243016ae76a32ce, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.11	0.12
query3	0.25	0.19	0.20
query4	1.59	0.19	0.11
query5	0.57	0.55	0.56
query6	1.21	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.58	0.54	0.50
query10	0.56	0.57	0.57
query11	0.16	0.11	0.10
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.79	0.80	0.83
query15	0.88	0.85	0.85
query16	0.38	0.39	0.40
query17	1.02	1.02	1.03
query18	0.22	0.21	0.21
query19	1.91	1.84	1.81
query20	0.02	0.01	0.01
query21	15.42	0.90	0.55
query22	0.76	1.17	0.64
query23	14.98	1.36	0.61
query24	6.79	1.52	1.41
query25	0.49	0.30	0.06
query26	0.60	0.15	0.15
query27	0.05	0.05	0.04
query28	9.16	0.91	0.45
query29	12.57	3.98	3.38
query30	0.25	0.09	0.07
query31	2.81	0.59	0.39
query32	3.23	0.54	0.46
query33	3.02	3.10	3.04
query34	15.57	5.15	4.50
query35	4.51	4.54	4.48
query36	0.68	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 102.51 s
Total hot run time: 29.78 s

zclllyybb · 2025-05-16T16:36:15Z

run buildall

zclllyybb · 2026-01-24T13:07:13Z

run buildall

doris-robot · 2026-01-24T13:44:04Z

TPC-DS: Total hot run time: 0 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3572829cf4940deff93b17329e47d968376303d4, data reload: false
query5 query6 query7 query8 query9 query10 query11 query12 query13 query14 query14_1 query15 query16 query17 query18 query19 query20 query21 query22 query23 query23_1 query24 query24_1 query25 query26 query27 query28 query29 query30 query31 query32 query33 query34 query35 query36 query37 query38 query39 query39_1 query40 query41 query42 query43 query44 query45 query46 query47 query48 query49 query50 query51 query52 query53 query54 query55 query56 query57 query58 query59 query60 query61 query62 query63 query64 query65 query66 query67 query68 query69 query70 query71 query72 query73 query74 query75 query76 query77 query78 query79 query80 query81 query82 query83 query84 query85 query86 query87 query88 query89 query90 query91 query92 query93 query94 query95 query96 query97 query98 query99 Total cold run time: 0 ms
Total hot run time: 0 ms

hello-stephen · 2026-01-24T14:33:27Z

BE UT Coverage Report

Increment line coverage 54.97% (199/362) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.73% (19221/36452)
Line Coverage	36.07% (178834/495782)
Region Coverage	32.55% (138387/425177)
Branch Coverage	33.47% (59876/178887)

hello-stephen · 2026-01-24T16:47:40Z

BE Regression && UT Coverage Report

Increment line coverage 59.28% (214/361) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.03% (20377/35731)
Line Coverage	39.85% (197370/495256)
Region Coverage	36.56% (157117/429708)
Branch Coverage	37.07% (66606/179662)

hello-stephen · 2026-01-24T16:59:13Z

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

zclllyybb · 2026-01-25T12:31:04Z

run buildall

doris-robot · 2026-01-25T14:17:50Z

TPC-H: Total hot run time: 30725 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a7988a74c94b7f13ccc32bb172bd5a714f47ba5, data reload: false
------ Round 1 ----------------------------------
q1	17638	4736	4558	4558
q2	2064	321	199	199
q3	10194	1273	733	733
q4	10202	760	304	304
q5	7530	2098	1809	1809
q6	237	168	141	141
q7	872	704	591	591
q8	9285	1353	1082	1082
q9	5163	4577	4614	4577
q10	6810	1674	1263	1263
q11	509	289	278	278
q12	387	379	220	220
q13	17790	3776	3057	3057
q14	245	230	206	206
q15	596	517	516	516
q16	656	635	586	586
q17	665	784	511	511
q18	6775	6532	6370	6370
q19	1481	980	629	629
q20	393	339	232	232
q21	2644	2083	1895	1895
q22	1033	1015	968	968
Total cold run time: 103169 ms
Total hot run time: 30725 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4842	4690	4680	4680
q2	328	384	317	317
q3	2162	2667	2262	2262
q4	1311	1778	1310	1310
q5	4172	4048	4258	4048
q6	189	165	127	127
q7	1917	1847	1911	1847
q8	2653	2574	2492	2492
q9	7264	7296	7184	7184
q10	2534	2746	2357	2357
q11	576	485	463	463
q12	760	767	612	612
q13	3603	4099	3415	3415
q14	340	308	288	288
q15	564	516	505	505
q16	639	678	625	625
q17	1187	1438	1422	1422
q18	8336	7838	7797	7797
q19	912	864	851	851
q20	2013	2041	1864	1864
q21	4398	4246	4121	4121
q22	1099	1049	964	964
Total cold run time: 51799 ms
Total hot run time: 49551 ms

doris-robot · 2026-01-25T14:28:36Z

TPC-DS: Total hot run time: 173230 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a7988a74c94b7f13ccc32bb172bd5a714f47ba5, data reload: false
query5	4406	653	504	504
query6	327	211	192	192
query7	4219	472	262	262
query8	334	250	230	230
query9	8726	2924	2919	2919
query10	455	315	312	312
query11	15688	15362	14909	14909
query12	179	116	114	114
query13	1231	451	363	363
query14	5557	3076	2770	2770
query14_1	2642	2592	2625	2592
query15	200	190	169	169
query16	977	477	475	475
query17	1067	646	560	560
query18	2455	445	339	339
query19	197	177	156	156
query20	122	126	116	116
query21	223	138	129	129
query22	3932	4165	4069	4069
query23	15879	15483	15209	15209
query23_1	15184	15395	15318	15318
query24	7216	1559	1156	1156
query24_1	1149	1168	1157	1157
query25	539	456	416	416
query26	1251	277	153	153
query27	2770	464	277	277
query28	4568	2160	2163	2160
query29	820	550	457	457
query30	321	255	214	214
query31	790	632	520	520
query32	87	79	81	79
query33	534	358	310	310
query34	897	892	559	559
query35	728	764	682	682
query36	885	923	821	821
query37	140	97	90	90
query38	2775	2716	2624	2624
query39	786	761	739	739
query39_1	701	704	699	699
query40	218	141	119	119
query41	77	69	68	68
query42	98	107	101	101
query43	438	444	430	430
query44	1327	758	755	755
query45	191	184	177	177
query46	836	948	589	589
query47	1348	1444	1396	1396
query48	321	333	245	245
query49	619	452	359	359
query50	668	284	208	208
query51	3815	3921	3807	3807
query52	93	93	83	83
query53	227	218	169	169
query54	303	273	266	266
query55	84	81	76	76
query56	307	325	317	317
query57	1042	1023	924	924
query58	283	276	260	260
query59	2049	2253	2083	2083
query60	341	343	317	317
query61	173	173	208	173
query62	406	349	327	327
query63	190	161	168	161
query64	5005	1138	837	837
query65	3894	3803	3621	3621
query66	1446	434	305	305
query67	15802	15812	15662	15662
query68	2390	1044	729	729
query69	384	315	278	278
query70	984	858	960	858
query71	299	287	265	265
query72	5309	3133	3387	3133
query73	598	723	317	317
query74	8633	8775	8449	8449
query75	2295	2369	1897	1897
query76	2270	1039	631	631
query77	347	375	310	310
query78	10026	9947	9271	9271
query79	2378	901	594	594
query80	1705	503	446	446
query81	548	279	249	249
query82	1012	151	117	117
query83	323	263	250	250
query84	262	111	96	96
query85	847	456	398	398
query86	404	278	291	278
query87	2922	2883	2725	2725
query88	3470	2596	2597	2596
query89	297	253	237	237
query90	1988	176	167	167
query91	162	154	128	128
query92	75	76	68	68
query93	1230	1016	648	648
query94	632	311	305	305
query95	584	329	373	329
query96	650	499	227	227
query97	2359	2368	2308	2308
query98	219	205	211	205
query99	615	576	528	528
Total cold run time: 248610 ms
Total hot run time: 173230 ms

doris-robot · 2026-01-25T14:33:37Z

ClickBench: Total hot run time: 26.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a7988a74c94b7f13ccc32bb172bd5a714f47ba5, data reload: false
query1	0.05	0.05	0.06
query2	0.11	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.12
query5	0.28	0.26	0.25
query6	1.14	0.64	0.66
query7	0.04	0.03	0.03
query8	0.06	0.04	0.05
query9	0.56	0.51	0.50
query10	0.54	0.54	0.55
query11	0.14	0.09	0.10
query12	0.15	0.11	0.10
query13	0.61	0.59	0.58
query14	0.96	0.94	0.94
query15	0.78	0.77	0.78
query16	0.41	0.39	0.38
query17	1.05	1.04	1.02
query18	0.23	0.21	0.21
query19	1.97	1.78	1.83
query20	0.02	0.01	0.01
query21	15.44	0.28	0.13
query22	4.98	0.06	0.05
query23	15.90	0.28	0.10
query24	1.28	0.29	0.58
query25	0.09	0.06	0.07
query26	0.15	0.14	0.14
query27	0.09	0.04	0.06
query28	4.25	1.08	0.88
query29	12.54	3.91	3.15
query30	0.28	0.14	0.12
query31	2.81	0.64	0.38
query32	3.24	0.55	0.46
query33	3.04	3.02	3.15
query34	16.48	5.04	4.42
query35	4.51	4.46	4.45
query36	0.64	0.49	0.48
query37	0.10	0.06	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.12
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 97.26 s
Total hot run time: 26.69 s

doris-robot · 2026-01-25T16:00:11Z

BE UT Coverage Report

Increment line coverage 54.55% (198/363) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.68% (19204/36452)
Line Coverage	36.03% (178635/495807)
Region Coverage	32.53% (138322/425254)
Branch Coverage	33.43% (59806/178907)

hello-stephen · 2026-01-25T17:06:35Z

BE Regression && UT Coverage Report

Increment line coverage 88.95% (322/362) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.46% (25533/35731)
Line Coverage	54.03% (267620/495278)
Region Coverage	51.46% (221144/429770)
Branch Coverage	52.97% (95175/179680)

hello-stephen · 2026-01-25T17:17:35Z

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

…hPartitioner (apache#48720) Related PR: bug introduced by apache#44102, expanded by apache#45256, partially fixed by apache#47911 Problem Summary: also fixed Auto Partition + Auto Increment-Column may core dump in loading

…abletSinkHashPartitioner (#48720) (#60280) pick #48720

…abletSinkHashPartitioner (apache#48720) (apache#60280) pick apache#48720

zclllyybb force-pushed the fix_resend_ap branch from 60151a6 to 0df083bCompare March 5, 2025 20:56

zclllyybb force-pushed the fix_resend_ap branch 4 times, most recently from 05c9422 to 5a86da5Compare March 5, 2025 21:07

yiguolei reviewed Mar 6, 2025
View reviewed changes

Gabriel39 requested changes Mar 6, 2025
View reviewed changes

Comment threadbe/src/vec/sink/tablet_sink_hash_partitioner.cpp Outdated

zclllyybb force-pushed the fix_resend_ap branch 2 times, most recently from a1aa63a to e74e9c5Compare May 14, 2025 07:36

zclllyybb force-pushed the fix_resend_ap branch from 0c6f154 to 5c6890fCompare May 16, 2025 16:23

zclllyybb force-pushed the fix_resend_ap branch from f03e812 to 3572829Compare January 24, 2026 13:07

fix comment
2a7988a

zclllyybb force-pushed the fix_resend_ap branch from 3572829 to 2a7988aCompare January 25, 2026 12:30

Mryange approved these changes Jan 26, 2026
View reviewed changes

yiguolei approved these changes Jan 27, 2026
View reviewed changes

zclllyybb merged commit 9571a91 into apache:masterJan 27, 2026
27 of 28 checks passed

zclllyybb deleted the fix_resend_ap branch January 27, 2026 09:14

zclllyybb added the approved Indicates a PR has been approved by one committer. label Jan 27, 2026

github-actionsBot added the dev/4.0.x-conflict label Jan 27, 2026

zclllyybb mentioned this pull request Jan 27, 2026
branch-4.0: [Refactor](shuffle) Fix wrong deal of batching block in TabletSinkHashPartitioner (#48720) #60280
Merged

yiguolei pushed a commit that referenced this pull request Jan 30, 2026
branch-4.0: [Refactor](shuffle) Fix wrong deal of batching block in T…
a283074
…abletSinkHashPartitioner (#48720) (#60280) pick #48720

yiguolei added dev/4.0.4-merged and removed dev/4.0.x dev/4.0.x-conflict labels Jan 30, 2026

zclllyybb mentioned this pull request Feb 18, 2026
[Bug] doris::vectorized::ColumnStr<unsigned int>::insert_indices_from #60713
Closed
3 tasks

ybtsdst pushed a commit to ybtsdst/doris that referenced this pull request Feb 27, 2026
branch-4.0: [Refactor](shuffle) Fix wrong deal of batching block in T…
5c6ff92
…abletSinkHashPartitioner (apache#48720) (apache#60280) pick apache#48720

Uh oh!

Conversation

zclllyybb commented Mar 5, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 5, 2025

Uh oh!

zclllyybb commented Mar 5, 2025

Uh oh!

zclllyybb commented Mar 5, 2025

Uh oh!

doris-robot commented Mar 5, 2025

Uh oh!

doris-robot commented Mar 5, 2025

Uh oh!

doris-robot commented Mar 5, 2025

Uh oh!

doris-robot commented Mar 5, 2025

BE UT Coverage Report

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zclllyybb commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

zclllyybb commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

zclllyybb commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

doris-robot commented May 14, 2025

Uh oh!

zclllyybb commented May 16, 2025

Uh oh!

zclllyybb commented Jan 24, 2026

Uh oh!

doris-robot commented Jan 24, 2026

Uh oh!

hello-stephen commented Jan 24, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 24, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jan 24, 2026

FE Regression Coverage Report

Uh oh!

zclllyybb commented Jan 25, 2026

Uh oh!

doris-robot commented Jan 25, 2026

Uh oh!

doris-robot commented Jan 25, 2026

Uh oh!

doris-robot commented Jan 25, 2026

Uh oh!

doris-robot commented Jan 25, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 25, 2026

BE Regression && UT Coverage Report

zclllyybb commented Mar 5, 2025•
edited
Loading