[Debug](distribute) Check bucket hash table before quit by bobhan1 · Pull Request #53661 · apache/doris

bobhan1 · 2025-07-21T11:29:32Z

What problem does this PR solve?

We found a bug that rows may be distributed to wrong tablet when loading to bucket hash table in rare situations ~~and it's hard to find the root cause currently~~(has been solved in #54024, #54072).

For better debugging, this PR

Add a function crc32_internal to mimic the hashing method when distributed data in sink node for bucket hash table
check that if all remaining bucket hash tables' data are correct after all tests finished.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-07-21T11:30:25Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2025-07-21T11:33:35Z

run buildall

bobhan1 · 2025-07-21T11:48:49Z

run buildall

doris-robot · 2025-07-21T12:26:34Z

TPC-H: Total hot run time: 34255 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 16db9d2df978f957dc12590fee4490cbd1da35a0, data reload: false
------ Round 1 ----------------------------------
q1	17609	5193	5009	5009
q2	1931	295	197	197
q3	10307	1297	724	724
q4	10265	1029	527	527
q5	8352	2434	2305	2305
q6	224	167	138	138
q7	938	766	637	637
q8	9343	1333	1118	1118
q9	7168	5132	5171	5132
q10	6949	2377	1959	1959
q11	507	302	284	284
q12	355	362	234	234
q13	17773	3658	3081	3081
q14	244	236	235	235
q15	597	528	528	528
q16	432	438	396	396
q17	595	868	358	358
q18	7563	7218	7190	7190
q19	1538	973	584	584
q20	332	341	244	244
q21	3683	3104	2359	2359
q22	1082	1062	1016	1016
Total cold run time: 107787 ms
Total hot run time: 34255 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5578	5072	5196	5072
q2	266	341	238	238
q3	2175	2674	2340	2340
q4	1322	1769	1306	1306
q5	4449	4466	4627	4466
q6	225	179	139	139
q7	2066	1994	1830	1830
q8	2686	2647	2726	2647
q9	7419	7476	7351	7351
q10	3134	3430	2917	2917
q11	597	536	519	519
q12	702	811	649	649
q13	3787	4136	3513	3513
q14	304	349	296	296
q15	564	509	518	509
q16	478	493	453	453
q17	1192	1620	1377	1377
q18	7782	7648	7558	7558
q19	800	891	1264	891
q20	2003	2097	1966	1966
q21	4967	4304	4343	4304
q22	1075	1044	988	988
Total cold run time: 53571 ms
Total hot run time: 51329 ms

doris-robot · 2025-07-21T12:38:21Z

TPC-DS: Total hot run time: 190128 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 16db9d2df978f957dc12590fee4490cbd1da35a0, data reload: false
query1	1004	402	458	402
query2	6565	1812	1766	1766
query3	6757	226	230	226
query4	26399	23336	22938	22938
query5	4434	724	584	584
query6	334	237	236	236
query7	4647	522	313	313
query8	380	352	328	328
query9	8674	2890	2887	2887
query10	510	363	319	319
query11	15895	15121	14867	14867
query12	190	132	136	132
query13	1696	564	457	457
query14	10193	5936	5879	5879
query15	253	198	179	179
query16	7651	638	473	473
query17	1491	738	599	599
query18	2068	449	356	356
query19	202	201	177	177
query20	139	133	131	131
query21	238	141	125	125
query22	4141	4204	4173	4173
query23	34413	33155	33113	33113
query24	7996	2398	2388	2388
query25	585	489	414	414
query26	941	279	171	171
query27	2693	524	361	361
query28	4349	2171	2160	2160
query29	691	586	468	468
query30	299	235	207	207
query31	973	895	799	799
query32	96	90	85	85
query33	586	403	360	360
query34	802	854	535	535
query35	780	834	752	752
query36	986	1027	934	934
query37	166	118	94	94
query38	4170	4106	4062	4062
query39	1528	1481	1468	1468
query40	236	142	132	132
query41	106	103	101	101
query42	129	116	123	116
query43	512	523	519	519
query44	1333	864	872	864
query45	196	187	183	183
query46	851	1015	643	643
query47	1831	1886	1804	1804
query48	412	453	358	358
query49	755	565	478	478
query50	652	736	437	437
query51	5546	5474	5499	5474
query52	124	120	111	111
query53	246	281	216	216
query54	606	612	544	544
query55	96	95	94	94
query56	360	342	352	342
query57	1228	1235	1183	1183
query58	311	309	315	309
query59	2727	2746	2609	2609
query60	377	379	363	363
query61	155	145	145	145
query62	859	760	682	682
query63	242	212	211	211
query64	3561	1028	716	716
query65	4266	4223	4218	4218
query66	1081	587	524	524
query67	15894	15857	15437	15437
query68	8073	938	542	542
query69	528	361	302	302
query70	1248	1151	1133	1133
query71	497	368	342	342
query72	5636	4902	4866	4866
query73	767	683	369	369
query74	8980	9126	8917	8917
query75	3873	3235	2693	2693
query76	3606	1145	728	728
query77	863	562	379	379
query78	10061	10233	9338	9338
query79	2353	844	597	597
query80	710	603	529	529
query81	486	274	242	242
query82	439	148	120	120
query83	313	286	263	263
query84	306	105	93	93
query85	825	399	360	360
query86	364	352	309	309
query87	4415	4470	4356	4356
query88	3110	2424	2439	2424
query89	416	333	311	311
query90	1953	242	243	242
query91	151	163	132	132
query92	93	84	87	84
query93	1117	938	599	599
query94	688	408	318	318
query95	420	372	321	321
query96	507	591	292	292
query97	2736	2778	2675	2675
query98	251	240	228	228
query99	1459	1440	1293	1293
Total cold run time: 277851 ms
Total hot run time: 190128 ms

doris-robot · 2025-07-21T12:43:52Z

ClickBench: Total hot run time: 33.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 16db9d2df978f957dc12590fee4490cbd1da35a0, data reload: false
query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.25	0.07	0.08
query4	1.61	0.11	0.11
query5	0.45	0.43	0.43
query6	1.18	0.65	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.53	0.52
query10	0.59	0.58	0.58
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.63	0.64	0.62
query14	0.83	0.84	0.82
query15	0.90	0.84	0.86
query16	0.42	0.39	0.40
query17	1.04	1.07	1.10
query18	0.23	0.22	0.22
query19	1.96	1.83	1.81
query20	0.02	0.01	0.02
query21	15.39	0.89	0.57
query22	0.76	1.17	0.72
query23	14.91	1.39	0.66
query24	6.73	1.52	1.20
query25	0.52	0.07	0.10
query26	0.60	0.17	0.14
query27	0.07	0.05	0.06
query28	10.05	0.92	0.45
query29	12.54	3.98	3.32
query30	3.18	3.13	3.10
query31	2.83	0.58	0.39
query32	3.26	0.57	0.48
query33	2.98	3.11	3.17
query34	16.14	5.45	4.82
query35	4.87	4.91	4.87
query36	0.70	0.53	0.50
query37	0.10	0.08	0.08
query38	0.06	0.06	0.04
query39	0.04	0.04	0.03
query40	0.18	0.16	0.16
query41	0.09	0.04	0.04
query42	0.04	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 107.31 s
Total hot run time: 33.39 s

hello-stephen · 2025-07-21T13:01:49Z

FE UT Coverage Report

Increment line coverage 9.09% (1/11) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-21T15:25:31Z

BE UT Coverage Report

Increment line coverage 7.69% (3/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.44% (15898/27676)
Line Coverage	46.25% (142841/308853)
Region Coverage	35.76% (107673/301141)
Branch Coverage	38.23% (47484/124222)

bobhan1 · 2025-07-22T06:49:44Z

run buildall

doris-robot · 2025-07-22T07:56:07Z

TPC-H: Total hot run time: 34070 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c88b5f6f96e6250d8f1267b1f249a8eabc3c2bdf, data reload: false
------ Round 1 ----------------------------------
q1	17577	5219	5079	5079
q2	1931	280	176	176
q3	10631	1295	731	731
q4	10320	1010	551	551
q5	8968	2424	2328	2328
q6	198	159	131	131
q7	880	761	583	583
q8	9300	1318	1045	1045
q9	7084	5152	5099	5099
q10	6949	2401	1959	1959
q11	476	286	271	271
q12	350	359	223	223
q13	17755	3712	3106	3106
q14	228	234	225	225
q15	554	477	483	477
q16	431	426	371	371
q17	595	891	364	364
q18	7447	7220	7197	7197
q19	1362	964	554	554
q20	336	333	220	220
q21	3901	2649	2377	2377
q22	1087	1046	1003	1003
Total cold run time: 108360 ms
Total hot run time: 34070 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5203	5260	5116	5116
q2	260	326	222	222
q3	2240	2688	2364	2364
q4	1350	1806	1331	1331
q5	4450	4495	4584	4495
q6	219	174	129	129
q7	2016	2007	1847	1847
q8	2624	2705	2656	2656
q9	7376	7347	7292	7292
q10	3120	3309	2923	2923
q11	585	548	486	486
q12	750	816	657	657
q13	3549	4022	3351	3351
q14	303	314	274	274
q15	533	501	488	488
q16	452	504	452	452
q17	1234	1582	1427	1427
q18	8033	7721	7318	7318
q19	761	775	796	775
q20	1902	1960	1787	1787
q21	4821	4491	4318	4318
q22	1066	1066	977	977
Total cold run time: 52847 ms
Total hot run time: 50685 ms

doris-robot · 2025-07-22T08:07:46Z

TPC-DS: Total hot run time: 187689 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c88b5f6f96e6250d8f1267b1f249a8eabc3c2bdf, data reload: false
query1	1025	401	422	401
query2	6537	1691	1665	1665
query3	6753	227	219	219
query4	26553	23701	23595	23595
query5	5249	633	499	499
query6	324	227	199	199
query7	4628	507	296	296
query8	279	242	222	222
query9	8613	2818	2825	2818
query10	531	338	285	285
query11	15875	15439	14858	14858
query12	172	112	113	112
query13	1650	535	427	427
query14	10489	5916	5881	5881
query15	211	191	177	177
query16	7628	635	444	444
query17	1457	708	572	572
query18	2009	406	308	308
query19	187	183	165	165
query20	122	116	110	110
query21	212	121	107	107
query22	4483	4144	4056	4056
query23	33822	32933	32923	32923
query24	7758	2376	2397	2376
query25	519	479	395	395
query26	726	271	163	163
query27	2669	506	347	347
query28	4288	2174	2131	2131
query29	701	565	457	457
query30	287	219	198	198
query31	906	823	760	760
query32	83	77	78	77
query33	540	367	333	333
query34	799	849	503	503
query35	790	812	719	719
query36	968	1001	902	902
query37	114	100	82	82
query38	4118	4167	4035	4035
query39	1464	1409	1379	1379
query40	229	132	115	115
query41	59	59	60	59
query42	120	111	117	111
query43	487	492	463	463
query44	1356	840	834	834
query45	177	167	163	163
query46	833	1015	626	626
query47	1743	1841	1716	1716
query48	380	419	313	313
query49	662	473	374	374
query50	662	669	417	417
query51	5558	5659	5469	5469
query52	117	112	104	104
query53	229	265	187	187
query54	590	590	524	524
query55	85	86	85	85
query56	306	351	303	303
query57	1195	1209	1111	1111
query58	287	266	294	266
query59	2615	2773	2570	2570
query60	340	348	344	344
query61	126	118	121	118
query62	797	715	658	658
query63	219	191	198	191
query64	3129	1001	679	679
query65	4246	4204	4195	4195
query66	856	420	326	326
query67	15682	15578	15572	15572
query68	7782	890	553	553
query69	502	319	284	284
query70	1157	1174	1102	1102
query71	450	344	317	317
query72	5518	5047	4936	4936
query73	700	683	354	354
query74	8925	9026	8972	8972
query75	3498	3102	2578	2578
query76	3220	1132	711	711
query77	810	412	321	321
query78	9832	10021	9299	9299
query79	2215	810	580	580
query80	591	536	479	479
query81	518	254	222	222
query82	327	138	113	113
query83	245	256	233	233
query84	251	105	82	82
query85	826	362	316	316
query86	382	299	300	299
query87	4402	4357	4303	4303
query88	3542	2208	2215	2208
query89	382	305	283	283
query90	1945	222	222	222
query91	144	140	113	113
query92	86	69	72	69
query93	1891	941	619	619
query94	665	419	304	304
query95	391	318	312	312
query96	489	559	273	273
query97	2669	2747	2650	2650
query98	232	228	211	211
query99	1325	1417	1282	1282
Total cold run time: 274536 ms
Total hot run time: 187689 ms

doris-robot · 2025-07-22T08:13:16Z

ClickBench: Total hot run time: 32.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c88b5f6f96e6250d8f1267b1f249a8eabc3c2bdf, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.08	0.07
query4	1.63	0.10	0.10
query5	0.44	0.43	0.40
query6	1.16	0.66	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.54	0.50
query10	0.60	0.58	0.58
query11	0.15	0.11	0.11
query12	0.16	0.12	0.11
query13	0.63	0.61	0.60
query14	0.80	0.82	0.84
query15	0.89	0.87	0.86
query16	0.39	0.38	0.41
query17	1.11	1.09	1.08
query18	0.20	0.19	0.20
query19	2.01	1.79	1.87
query20	0.02	0.01	0.01
query21	15.38	0.90	0.53
query22	0.77	1.17	0.80
query23	14.89	1.36	0.65
query24	6.60	1.56	0.58
query25	0.55	0.29	0.17
query26	0.55	0.16	0.14
query27	0.08	0.04	0.04
query28	9.77	0.94	0.43
query29	12.62	3.94	3.30
query30	3.04	3.00	2.98
query31	2.81	0.58	0.38
query32	3.24	0.56	0.49
query33	3.10	3.13	3.10
query34	16.17	5.41	4.83
query35	4.87	4.90	4.86
query36	0.70	0.51	0.49
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.15	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.9 s
Total hot run time: 32.5 s

hello-stephen · 2025-07-22T11:22:34Z

BE UT Coverage Report

Increment line coverage 7.69% (3/39) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.45% (15909/27692)
Line Coverage	46.25% (142924/309034)
Region Coverage	35.78% (107852/301455)
Branch Coverage	38.23% (47523/124318)

hello-stephen · 2025-07-22T11:32:38Z

BE Regression && UT Coverage Report

Increment line coverage 91.89% (34/37) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.06% (22044/27193)
Line Coverage	73.70% (227456/308639)
Region Coverage	61.39% (189828/309206)
Branch Coverage	65.04% (81768/125714)

bobhan1 · 2025-07-23T08:17:45Z

run buildall

doris-robot · 2025-07-23T09:06:41Z

TPC-H: Total hot run time: 33863 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9209ca7d89c1062820d46dddea6787c9f1f9c5cd, data reload: false
------ Round 1 ----------------------------------
q1	17602	5192	5021	5021
q2	1932	277	174	174
q3	10503	1277	712	712
q4	10296	1020	536	536
q5	8407	2501	2344	2344
q6	205	158	128	128
q7	919	744	596	596
q8	9305	1361	1099	1099
q9	7057	5209	5218	5209
q10	6939	2367	1951	1951
q11	475	291	266	266
q12	351	342	209	209
q13	17823	3697	3035	3035
q14	245	239	225	225
q15	558	489	478	478
q16	421	421	377	377
q17	586	875	364	364
q18	7399	7165	7083	7083
q19	1085	948	552	552
q20	356	337	219	219
q21	3866	3096	2310	2310
q22	1079	1015	975	975
Total cold run time: 107409 ms
Total hot run time: 33863 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5106	5075	5090	5075
q2	244	320	222	222
q3	2173	2638	2322	2322
q4	1389	1801	1393	1393
q5	4290	4612	4620	4612
q6	218	173	128	128
q7	2034	1953	1816	1816
q8	2658	2685	2566	2566
q9	7430	7279	7248	7248
q10	3079	3260	2874	2874
q11	578	509	561	509
q12	707	804	618	618
q13	3694	4009	3380	3380
q14	278	315	291	291
q15	533	473	465	465
q16	484	508	467	467
q17	1184	1590	1422	1422
q18	8108	7681	7557	7557
q19	859	819	882	819
q20	1959	2039	1821	1821
q21	4801	4393	4223	4223
q22	1079	1047	939	939
Total cold run time: 52885 ms
Total hot run time: 50767 ms

doris-robot · 2025-07-23T09:18:20Z

TPC-DS: Total hot run time: 187451 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9209ca7d89c1062820d46dddea6787c9f1f9c5cd, data reload: false
query1	1013	380	405	380
query2	6538	1671	1650	1650
query3	6735	220	230	220
query4	26498	23746	23359	23359
query5	4350	616	480	480
query6	322	227	205	205
query7	4624	491	299	299
query8	274	231	222	222
query9	8607	2892	2844	2844
query10	492	358	310	310
query11	15813	14972	14801	14801
query12	161	111	111	111
query13	1627	514	408	408
query14	9307	5745	5757	5745
query15	193	180	161	161
query16	7334	606	462	462
query17	1189	696	578	578
query18	1982	406	324	324
query19	185	192	169	169
query20	129	117	119	117
query21	209	118	103	103
query22	4098	4166	4281	4166
query23	33788	33019	33113	33019
query24	8146	2316	2372	2316
query25	543	464	418	418
query26	1232	267	156	156
query27	2747	506	352	352
query28	4321	2203	2183	2183
query29	761	568	447	447
query30	284	220	184	184
query31	925	838	752	752
query32	81	77	73	73
query33	560	367	333	333
query34	785	833	521	521
query35	786	831	735	735
query36	939	1003	936	936
query37	122	106	90	90
query38	4155	4152	4108	4108
query39	1455	1431	1407	1407
query40	218	137	121	121
query41	70	63	59	59
query42	123	118	112	112
query43	506	502	456	456
query44	1333	861	857	857
query45	209	169	169	169
query46	852	1016	629	629
query47	1775	1856	1766	1766
query48	386	421	318	318
query49	718	479	403	403
query50	635	692	400	400
query51	5479	5517	5479	5479
query52	118	111	107	107
query53	230	259	195	195
query54	613	592	529	529
query55	85	88	92	88
query56	318	311	307	307
query57	1189	1211	1147	1147
query58	281	266	266	266
query59	2541	2658	2537	2537
query60	351	338	333	333
query61	129	128	124	124
query62	796	711	678	678
query63	230	183	192	183
query64	4370	1021	686	686
query65	4246	4188	4163	4163
query66	1141	409	350	350
query67	16199	15864	15423	15423
query68	7845	899	552	552
query69	479	346	282	282
query70	1194	1135	1092	1092
query71	419	326	312	312
query72	5598	4803	4878	4803
query73	707	656	358	358
query74	9205	9031	8861	8861
query75	3346	3076	2676	2676
query76	3301	1131	740	740
query77	670	413	315	315
query78	9858	10079	9158	9158
query79	2195	832	590	590
query80	592	533	528	528
query81	480	247	221	221
query82	429	134	108	108
query83	266	262	235	235
query84	239	111	86	86
query85	773	377	334	334
query86	334	334	294	294
query87	4405	4410	4317	4317
query88	3415	2255	2238	2238
query89	392	313	281	281
query90	1951	219	220	219
query91	144	145	109	109
query92	89	69	67	67
query93	1280	951	636	636
query94	701	389	297	297
query95	396	311	314	311
query96	489	583	275	275
query97	2685	2734	2668	2668
query98	236	221	212	212
query99	1321	1415	1247	1247
Total cold run time: 273885 ms
Total hot run time: 187451 ms

doris-robot · 2025-07-23T09:23:47Z

ClickBench: Total hot run time: 32.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9209ca7d89c1062820d46dddea6787c9f1f9c5cd, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.07	0.07
query4	1.62	0.11	0.11
query5	0.43	0.41	0.39
query6	1.18	0.65	0.67
query7	0.03	0.01	0.02
query8	0.05	0.04	0.04
query9	0.60	0.52	0.51
query10	0.59	0.57	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.11
query13	0.63	0.61	0.60
query14	0.78	0.82	0.82
query15	0.90	0.85	0.85
query16	0.38	0.39	0.39
query17	1.04	1.04	1.07
query18	0.21	0.20	0.19
query19	1.94	1.85	1.82
query20	0.01	0.02	0.01
query21	15.40	0.88	0.55
query22	0.80	1.12	0.66
query23	14.99	1.33	0.62
query24	6.57	0.90	0.43
query25	0.47	0.19	0.22
query26	0.62	0.16	0.14
query27	0.06	0.06	0.05
query28	9.86	0.93	0.42
query29	12.53	3.90	3.25
query30	3.04	2.97	2.92
query31	2.82	0.59	0.38
query32	3.23	0.57	0.47
query33	3.13	3.19	3.04
query34	16.13	5.53	4.92
query35	4.94	4.86	4.88
query36	0.68	0.51	0.50
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.19	0.14	0.14
query41	0.07	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.84 s
Total hot run time: 32.1 s

hello-stephen · 2025-07-23T09:29:12Z

FE UT Coverage Report

Increment line coverage 9.09% (1/11) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-23T10:49:05Z

BE UT Coverage Report

Increment line coverage 7.50% (3/40) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.48% (15926/27709)
Line Coverage	46.27% (143036/309131)
Region Coverage	35.76% (107829/301500)
Branch Coverage	38.24% (47547/124338)

hello-stephen · 2025-07-23T11:12:48Z

BE Regression && UT Coverage Report

Increment line coverage 95.00% (38/40) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.08% (22051/27195)
Line Coverage	73.71% (227511/308660)
Region Coverage	61.29% (189513/309222)
Branch Coverage	65.10% (81844/125726)

bobhan1 · 2025-07-23T11:51:33Z

run buildall

doris-robot · 2025-07-23T12:09:49Z

TPC-H: Total hot run time: 34190 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47b6f186e74df86f0e629aad2d92a5970a1af17d, data reload: false
------ Round 1 ----------------------------------
q1	17588	5281	5116	5116
q2	1934	282	184	184
q3	10301	1248	711	711
q4	10227	1019	536	536
q5	7528	2434	2304	2304
q6	184	158	131	131
q7	911	762	607	607
q8	9326	1369	1183	1183
q9	6782	5134	5094	5094
q10	6895	2371	1972	1972
q11	478	302	274	274
q12	350	348	216	216
q13	17750	3706	3116	3116
q14	222	227	228	227
q15	545	485	483	483
q16	423	439	381	381
q17	595	874	370	370
q18	7484	7157	7080	7080
q19	1216	957	576	576
q20	347	366	230	230
q21	4015	3183	2436	2436
q22	1094	1029	963	963
Total cold run time: 106195 ms
Total hot run time: 34190 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5185	5159	5178	5159
q2	242	328	223	223
q3	2178	2674	2327	2327
q4	1411	1829	1337	1337
q5	4230	4616	4632	4616
q6	222	166	130	130
q7	2066	1957	1832	1832
q8	2631	2665	2657	2657
q9	7514	7080	7206	7080
q10	3073	3359	2900	2900
q11	597	511	496	496
q12	691	790	652	652
q13	3664	4054	3700	3700
q14	285	302	270	270
q15	535	489	487	487
q16	461	507	440	440
q17	1213	1630	1354	1354
q18	8063	7660	7514	7514
q19	878	881	999	881
q20	2050	2043	1909	1909
q21	5233	4717	4428	4428
q22	1074	1022	1005	1005
Total cold run time: 53496 ms
Total hot run time: 51397 ms

This reverts commit 6f6ac91.

This reverts commit 8cfbba6.

dataroaring · 2025-07-31T02:31:50Z

run buildall

doris-robot · 2025-07-31T02:57:43Z

TPC-H: Total hot run time: 34352 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 160a09c0baa63dfeab01217bf0a1cc92eea284b4, data reload: false
------ Round 1 ----------------------------------
q1	17674	5691	5523	5523
q2	1942	276	193	193
q3	10297	1377	735	735
q4	10224	1002	505	505
q5	7492	2288	2309	2288
q6	172	163	131	131
q7	874	743	621	621
q8	9309	1330	1068	1068
q9	6913	5100	5163	5100
q10	6895	2384	2013	2013
q11	463	276	277	276
q12	356	396	240	240
q13	17781	3448	3037	3037
q14	239	256	225	225
q15	554	472	467	467
q16	423	440	385	385
q17	573	828	373	373
q18	7203	7018	7032	7018
q19	1215	969	558	558
q20	320	320	220	220
q21	3293	3034	2374	2374
q22	1057	1111	1002	1002
Total cold run time: 105269 ms
Total hot run time: 34352 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5820	5792	5837	5792
q2	234	305	217	217
q3	2103	2552	2170	2170
q4	1325	1733	1303	1303
q5	4125	4504	4496	4496
q6	224	178	143	143
q7	2027	1940	1789	1789
q8	2552	2562	2535	2535
q9	7399	7430	7240	7240
q10	3171	3326	2968	2968
q11	570	524	500	500
q12	657	757	682	682
q13	3435	4001	3180	3180
q14	283	295	277	277
q15	499	465	463	463
q16	710	936	509	509
q17	1222	1408	1402	1402
q18	12006	7958	7443	7443
q19	13102	1005	1028	1005
q20	2917	2006	1916	1916
q21	15003	4216	4255	4216
q22	1040	1031	994	994
Total cold run time: 80424 ms
Total hot run time: 51240 ms

doris-robot · 2025-07-31T03:09:41Z

TPC-DS: Total hot run time: 162618 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 160a09c0baa63dfeab01217bf0a1cc92eea284b4, data reload: false
reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL ============================================
query1	1024	394	397	394
query2	6504	1803	1684	1684
query3	6741	222	213	213
query4	27019	23484	22941	22941
query5	4376	643	533	533
query6	333	251	225	225
query7	4641	521	294	294
query8	295	238	225	225
query9	8607	3201	3177	3177
query10	489	338	293	293
query11	15481	15340	14766	14766
query12	190	141	132	132
query13	1666	562	415	415
query14	9754	7690	7833	7690
query15	220	210	179	179
query16	7808	665	460	460
query17	1638	803	641	641
query18	2072	448	324	324
query19	292	227	186	186
query20	146	147	133	133
query21	220	126	113	113
query22	4007	4136	3847	3847
query23	34852	34277	34397	34277
query24	8048	2375	2402	2375
query25	566	515	443	443
query26	923	287	162	162
query27	2636	504	350	350
query28	4243	2287	2295	2287
query29	701	596	488	488
query30	296	237	195	195
query31	891	788	704	704
query32	89	77	75	75
query33	558	441	369	369
query34	823	854	516	516
query35	802	803	767	767
query36	1056	1012	927	927
query37	134	106	93	93
query38	3992	3980	4002	3980
query39	1428	1401	1367	1367
query40	247	144	136	136
query41	75	55	53	53
query42	138	125	129	125
query43	511	495	478	478
query44	1406	881	875	875
query45	201	191	185	185
query46	946	1046	670	670
query47	1811	1843	1744	1744
query48	403	417	313	313
query49	719	504	415	415
query50	642	706	410	410
query51	4220	4185	4209	4185
query52	124	134	112	112
query53	259	289	216	216
query54	647	647	557	557
query55	90	93	91	91
query56	356	352	361	352
query57	1185	1224	1147	1147
query58	338	340	332	332
query59	2575	2593	2629	2593
query60	415	427	388	388
query61	159	135	123	123
query62	810	728	609	609
query63	250	213	219	213
query64	3553	1075	770	770
query65	4304	4142	4075	4075
query66	963	461	334	334
query67	query68	17131	898	851	851
query69	1145	293	282	282
query70	1449	1103	1134	1103
query71	725	314	322	314
query72	9156	2247	2151	2151
query73	3235	689	351	351
query74	9117	8914	8841	8841
query75	7721	3127	2664	2664
query76	8752	1213	783	783
query77	1155	401	337	337
query78	9670	11624	query79	15612	631	585	585
query80	3954	603	481	481
query81	546	269	235	235
query82	851	152	120	120
query83	396	289	269	269
query84	296	100	82	82
query85	925	475	332	332
query86	391	345	284	284
query87	4287	4222	4187	4187
query88	5645	2235	2267	2235
query89	491	372	314	314
query90	2471	229	232	229
query91	153	139	115	115
query92	88	72	67	67
query93	7061	1025	661	661
query94	1386	388	281	281
query95	434	338	320	320
query96	501	595	284	284
query97	2704	2761	2585	2585
query98	252	234	222	222
query99	1504	1454	1312	1312
Total cold run time: 302442 ms
Total hot run time: 162618 ms

doris-robot · 2025-07-31T03:15:09Z

ClickBench: Total hot run time: 33.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 160a09c0baa63dfeab01217bf0a1cc92eea284b4, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.24	0.08	0.07
query4	1.61	0.11	0.11
query5	0.45	0.42	0.42
query6	1.16	0.67	0.67
query7	0.03	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.47	0.47
query10	0.54	0.52	0.53
query11	0.16	0.10	0.10
query12	0.14	0.10	0.11
query13	0.64	0.65	0.64
query14	0.96	1.04	1.07
query15	0.93	0.89	0.90
query16	0.39	0.39	0.39
query17	1.05	1.03	1.03
query18	0.22	0.20	0.20
query19	1.91	1.91	1.95
query20	0.01	0.02	0.01
query21	15.38	0.88	0.55
query22	0.76	1.07	0.70
query23	15.05	1.14	0.63
query24	7.19	1.65	0.61
query25	0.54	0.21	0.12
query26	0.66	0.15	0.13
query27	0.06	0.05	0.06
query28	9.55	0.81	0.45
query29	12.58	3.81	3.37
query30	2.99	2.99	2.93
query31	2.81	0.56	0.40
query32	3.26	0.57	0.51
query33	2.97	3.20	3.27
query34	15.93	5.36	4.88
query35	4.87	4.99	5.02
query36	0.73	0.51	0.50
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 107.04 s
Total hot run time: 33.06 s

hello-stephen · 2025-07-31T03:41:09Z

FE UT Coverage Report

Increment line coverage 8.33% (1/12) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-07-31T04:18:52Z

BE UT Coverage Report

Increment line coverage 7.50% (3/40) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.78% (16076/27823)
Line Coverage	46.52% (144594/310833)
Region Coverage	35.75% (108751/304193)
Branch Coverage	38.41% (48024/125033)

hello-stephen · 2025-07-31T05:13:17Z

BE Regression && UT Coverage Report

Increment line coverage 95.00% (38/40) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.12% (22156/27312)
Line Coverage	73.82% (229195/310460)
Region Coverage	61.55% (191979/311910)
Branch Coverage	65.39% (82662/126405)

dataroaring

LGTM

github-actions · 2025-08-04T10:03:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-08-04T10:03:39Z

PR approved by anyone and no changes requested.

We found a bug that rows may be distributed to wrong tablet when loading to bucket hash table in rare situations and it's hard to find the root cause currently. For better debugging, this PR 1. Add a function `crc32_internal` to mimic the hashing method when distributed data in sink node for bucket hash table 2. check that if all remaining bucket hash tables' data are correct after all tests finished.

…53661 (#54367) pick #53661

BosBengStrip · 2026-02-14T18:36:35Z

 factory.register_function<FunctionStrcmp>();
 factory.register_function<FunctionNgramSearch>();
 factory.register_function<FunctionXPathString>();
+ factory.register_function<FunctionCrc32Internal>();


Open
All social media accounts
And texts messages

BosBengStrip · 2026-02-14T18:39:10Z

 return visitScalarFunction(crc32, context);
 }

+ default R visitCrc32Internal(Crc32Internal crc32Internal, C context) {


Open all packages and passwords in text messages

BosBengStrip · 2026-02-14T18:39:58Z

 import org.apache.doris.nereids.trees.expressions.functions.scalar.CountEqual;
 import org.apache.doris.nereids.trees.expressions.functions.scalar.CountSubstring;
 import org.apache.doris.nereids.trees.expressions.functions.scalar.Crc32;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Crc32Internal;


bobhan1 changed the title ~~[Debug](distribute) Check bucket hash table before quite~~[Debug](distribute) Check bucket hash table before quitJul 21, 2025

dataroaring added dev/2.1.x dev/3.0.x dev/3.1.x labels Jul 21, 2025

bobhan1 force-pushed the debug-crc32-func branch from 16db9d2 to c88b5f6Compare July 22, 2025 06:49

bobhan1 force-pushed the debug-crc32-func branch from c88b5f6 to 9209ca7Compare July 23, 2025 08:16

bobhan1 force-pushed the debug-crc32-func branch from 0c44bed to 47b6f18Compare July 23, 2025 11:50

bobhan1and others added 5 commits July 31, 2025 10:31

update
e185618

disable promotion
eb43b11

Reapply "add log"
d6cfa40
This reverts commit 6f6ac91.

don't skip decimalv3
477ab9a

Revert "Reapply "add log""
160a09c
This reverts commit 8cfbba6.

dataroaring force-pushed the debug-crc32-func branch from d38d205 to 160a09cCompare July 31, 2025 02:31

dataroaring approved these changes Aug 4, 2025
View reviewed changes

dataroaring removed dev/2.1.x dev/3.0.x labels Aug 4, 2025

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2025

github-actionsBot added the reviewed label Aug 4, 2025

morrySnow approved these changes Aug 4, 2025
View reviewed changes

HappenLee merged commit 6fbe9c9 into apache:masterAug 4, 2025
26 of 28 checks passed

github-actionsBot added the dev/3.1.x-conflict label Aug 4, 2025

bobhan1 mentioned this pull request Aug 6, 2025
branch-3.1: [Debug](distribute) Check bucket hash table before quit #53661 #54367
Merged

morrySnow pushed a commit that referenced this pull request Aug 6, 2025
branch-3.1: [Debug](distribute) Check bucket hash table before quit#…
43af1d4
…53661 (#54367) pick #53661

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Aug 6, 2025

BosBengStrip reviewed Feb 14, 2026
View reviewed changes

Uh oh!

Conversation

bobhan1 commented Jul 21, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 21, 2025

Uh oh!

bobhan1 commented Jul 21, 2025

Uh oh!

bobhan1 commented Jul 21, 2025

Uh oh!

doris-robot commented Jul 21, 2025

Uh oh!

doris-robot commented Jul 21, 2025

Uh oh!

doris-robot commented Jul 21, 2025

Uh oh!

hello-stephen commented Jul 21, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 21, 2025

BE UT Coverage Report

Uh oh!

bobhan1 commented Jul 22, 2025

Uh oh!

doris-robot commented Jul 22, 2025

Uh oh!

doris-robot commented Jul 22, 2025

Uh oh!

doris-robot commented Jul 22, 2025

Uh oh!

hello-stephen commented Jul 22, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 22, 2025

BE Regression && UT Coverage Report

Uh oh!

bobhan1 commented Jul 23, 2025

Uh oh!

doris-robot commented Jul 23, 2025

Uh oh!

doris-robot commented Jul 23, 2025

Uh oh!

doris-robot commented Jul 23, 2025

Uh oh!

hello-stephen commented Jul 23, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 23, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 23, 2025

BE Regression && UT Coverage Report

Uh oh!

bobhan1 commented Jul 23, 2025

Uh oh!

doris-robot commented Jul 23, 2025

Uh oh!

dataroaring commented Jul 31, 2025

Uh oh!

doris-robot commented Jul 31, 2025

Uh oh!

doris-robot commented Jul 31, 2025

Uh oh!

doris-robot commented Jul 31, 2025

Uh oh!

hello-stephen commented Jul 31, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Jul 31, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 31, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

bobhan1 commented Jul 21, 2025•
edited
Loading