branch-3.0: [fix](auto bucket)Fix auto bucket calc bucketnum err when partition size is invalid #52801 by deardeng · Pull Request #53250 · apache/doris

deardeng · 2025-07-15T02:54:06Z

cherry pick from #52801

…ize is invalid (apache#52801) …ize invalid 1. Fixed the problem that auto bucket will calculate wrong results when partition size is inaccurate - If `replica.size == 0`, filter out this replica. In the tablet.getDataSize function, the size is calculated by taking the average value of the replicas. When the size of a replica is 0, it will have a great impact on the average value. Therefore, the replicas with size=0 are filtered out. - If the partition size equals 0, do not include it in the estimation of the partition size. - If all versions with data partitions have sizes equal to 0, then the newly calculated bucket number for the partition will equal the bucket number of the previous version with a size greater than 0. Since we do not know the partition size of the data partitions (as stats thread have not been collected yet), we assume that the new partition's size equals the size of the previous version with a size greater than 0. Consequently, the bucket number will naturally equal that of the previous partition. 2. Added alarm log when the bucket num calculated by auto bucket exceeds the threshold

deardeng · 2025-07-15T02:54:08Z

run buildall

Thearas · 2025-07-15T02:54:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

doris-robot · 2025-07-15T03:28:20Z

TPC-H: Total hot run time: 39313 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b4930991d434ba9c1381e14c3b813d23ca0d339, data reload: false
------ Round 1 ----------------------------------
q1	17606	6788	6605	6605
q2	2092	192	162	162
q3	10597	1117	1128	1117
q4	10275	770	773	770
q5	7782	2845	2668	2668
q6	210	133	130	130
q7	981	628	595	595
q8	9585	1911	2008	1911
q9	7826	6400	6382	6382
q10	7044	2210	2265	2210
q11	458	254	257	254
q12	386	212	221	212
q13	17762	2935	2970	2935
q14	228	202	205	202
q15	530	473	460	460
q16	469	368	376	368
q17	957	533	520	520
q18	7256	6658	6516	6516
q19	1431	1120	1098	1098
q20	486	193	207	193
q21	3878	3168	3021	3021
q22	1117	984	1006	984
Total cold run time: 108956 ms
Total hot run time: 39313 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6648	6589	6617	6589
q2	325	240	232	232
q3	2893	2915	2929	2915
q4	2081	1864	1842	1842
q5	5668	5715	5730	5715
q6	212	130	128	128
q7	2218	1792	1823	1792
q8	3333	3543	3547	3543
q9	8734	8828	8782	8782
q10	3558	3524	3516	3516
q11	587	501	496	496
q12	798	595	578	578
q13	9369	3222	3210	3210
q14	308	264	273	264
q15	512	462	462	462
q16	497	437	436	436
q17	1835	1627	1597	1597
q18	8188	7690	7680	7680
q19	1679	1597	1576	1576
q20	2110	1858	1859	1858
q21	5235	5094	5061	5061
q22	1154	1079	1024	1024
Total cold run time: 67942 ms
Total hot run time: 59296 ms

doris-robot · 2025-07-15T03:40:38Z

TPC-DS: Total hot run time: 196358 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b4930991d434ba9c1381e14c3b813d23ca0d339, data reload: false
query1	1278	930	895	895
query2	6246	1866	1829	1829
query3	10850	4389	4346	4346
query4	33195	23862	23485	23485
query5	4453	456	438	438
query6	283	173	179	173
query7	3995	309	321	309
query8	272	213	222	213
query9	9478	2570	2563	2563
query10	486	262	253	253
query11	17967	15121	15180	15121
query12	154	101	100	100
query13	1550	421	426	421
query14	8648	6617	7353	6617
query15	258	179	183	179
query16	8016	484	474	474
query17	1619	583	609	583
query18	2126	310	305	305
query19	225	164	157	157
query20	126	114	118	114
query21	204	102	103	102
query22	4740	4612	4542	4542
query23	34997	34616	34216	34216
query24	11266	2938	2865	2865
query25	686	400	402	400
query26	1421	175	181	175
query27	2512	345	349	345
query28	7714	2165	2161	2161
query29	831	479	434	434
query30	268	162	159	159
query31	1055	805	849	805
query32	87	51	54	51
query33	763	311	292	292
query34	1122	527	521	521
query35	886	744	736	736
query36	1105	969	961	961
query37	144	75	68	68
query38	4025	3960	3871	3871
query39	1520	1483	1494	1483
query40	216	110	97	97
query41	48	47	47	47
query42	112	100	104	100
query43	523	486	477	477
query44	1237	842	818	818
query45	182	169	174	169
query46	1164	739	716	716
query47	2027	1930	1954	1930
query48	461	370	392	370
query49	947	403	402	402
query50	844	420	440	420
query51	7404	7273	7379	7273
query52	106	89	92	89
query53	262	184	182	182
query54	1032	470	469	469
query55	87	77	76	76
query56	264	259	246	246
query57	1296	1200	1174	1174
query58	228	216	208	208
query59	3301	3035	3011	3011
query60	300	264	275	264
query61	133	132	135	132
query62	887	699	679	679
query63	221	193	188	188
query64	4175	753	738	738
query65	3498	3300	3224	3224
query66	1235	310	298	298
query67	15794	15754	15498	15498
query68	4449	584	572	572
query69	444	268	273	268
query70	1161	1105	1133	1105
query71	337	261	261	261
query72	6651	4075	3932	3932
query73	744	355	352	352
query74	10163	9299	8924	8924
query75	3381	2616	2668	2616
query76	2694	975	1066	975
query77	385	274	261	261
query78	10496	9627	9598	9598
query79	2067	601	602	601
query80	1051	429	424	424
query81	545	218	219	218
query82	892	95	86	86
query83	243	143	145	143
query84	233	84	85	84
query85	1313	306	291	291
query86	429	306	287	287
query87	4392	4261	4321	4261
query88	3889	2420	2381	2381
query89	417	300	288	288
query90	2025	187	185	185
query91	183	147	145	145
query92	58	48	48	48
query93	2402	559	557	557
query94	892	311	296	296
query95	362	258	261	258
query96	621	288	295	288
query97	3307	3178	3138	3138
query98	221	199	196	196
query99	1492	1299	1262	1262
Total cold run time: 302494 ms
Total hot run time: 196358 ms

doris-robot · 2025-07-15T03:46:01Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9b4930991d434ba9c1381e14c3b813d23ca0d339, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.53	0.53	0.52
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.55	0.50	0.50
query10	0.55	0.56	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.58
query14	0.76	0.81	0.80
query15	0.85	0.82	0.81
query16	0.37	0.37	0.40
query17	1.06	1.05	1.07
query18	0.24	0.22	0.21
query19	1.93	1.85	1.86
query20	0.01	0.02	0.01
query21	15.40	0.60	0.57
query22	2.63	2.13	1.60
query23	16.90	1.09	0.84
query24	2.94	1.69	1.76
query25	0.24	0.22	0.06
query26	0.50	0.14	0.13
query27	0.04	0.03	0.03
query28	9.10	0.54	0.46
query29	12.60	3.29	3.21
query30	0.24	0.06	0.06
query31	2.86	0.39	0.39
query32	3.23	0.46	0.46
query33	3.01	2.93	3.02
query34	16.98	4.50	4.50
query35	4.49	4.56	4.44
query36	0.68	0.49	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.19 s
Total hot run time: 30.62 s

deardeng · 2025-07-15T10:22:50Z

run buildall

doris-robot · 2025-07-15T10:42:14Z

TPC-H: Total hot run time: 39950 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d98e50753b2a23d4545cdaa3f2e9af1f5fe74be4, data reload: false
------ Round 1 ----------------------------------
q1	17581	6804	6598	6598
q2	2083	201	167	167
q3	10555	1134	1186	1134
q4	10202	760	711	711
q5	7741	2932	2728	2728
q6	215	135	136	135
q7	976	618	604	604
q8	9364	1945	2035	1945
q9	6728	6383	6421	6383
q10	6942	2262	2304	2262
q11	454	274	279	274
q12	404	222	222	222
q13	17803	3039	3011	3011
q14	236	205	209	205
q15	529	481	471	471
q16	494	392	393	392
q17	994	598	587	587
q18	7493	6806	6755	6755
q19	1399	1061	1042	1042
q20	484	211	218	211
q21	3993	3140	3278	3140
q22	1106	976	973	973
Total cold run time: 107776 ms
Total hot run time: 39950 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6569	6557	6878	6557
q2	336	246	232	232
q3	2943	2903	2978	2903
q4	2027	1854	1897	1854
q5	5713	5754	5764	5754
q6	208	131	133	131
q7	2232	1764	1836	1764
q8	3384	3542	3555	3542
q9	8797	9006	8866	8866
q10	3552	3519	3515	3515
q11	588	505	497	497
q12	800	597	621	597
q13	9948	3146	3167	3146
q14	312	265	261	261
q15	510	466	476	466
q16	499	452	432	432
q17	1831	1640	1596	1596
q18	8205	7799	7607	7607
q19	1666	1478	1583	1478
q20	2158	1892	1896	1892
q21	5322	5019	4994	4994
q22	1146	1059	998	998
Total cold run time: 68746 ms
Total hot run time: 59082 ms

doris-robot · 2025-07-15T10:54:35Z

TPC-DS: Total hot run time: 196881 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d98e50753b2a23d4545cdaa3f2e9af1f5fe74be4, data reload: false
query1	1266	904	885	885
query2	6225	1886	1871	1871
query3	10823	4339	4355	4339
query4	33063	23527	23723	23527
query5	3662	459	443	443
query6	259	172	179	172
query7	4002	313	327	313
query8	293	218	217	217
query9	9564	2568	2553	2553
query10	476	266	264	264
query11	18026	15177	15228	15177
query12	161	102	105	102
query13	1559	431	431	431
query14	9452	6728	7209	6728
query15	257	183	182	182
query16	8049	470	488	470
query17	1601	591	580	580
query18	2140	307	302	302
query19	211	165	167	165
query20	124	114	106	106
query21	209	106	103	103
query22	4641	4461	4434	4434
query23	34705	34086	33865	33865
query24	11279	2985	2999	2985
query25	691	435	427	427
query26	1772	178	175	175
query27	2770	355	365	355
query28	7698	2152	2181	2152
query29	1004	473	468	468
query30	264	162	164	162
query31	1071	813	850	813
query32	103	60	60	60
query33	767	308	311	308
query34	959	511	527	511
query35	897	729	725	725
query36	1122	951	955	951
query37	127	71	65	65
query38	4105	3954	3934	3934
query39	1533	1469	1498	1469
query40	269	109	106	106
query41	53	51	50	50
query42	115	103	99	99
query43	522	481	477	477
query44	1283	844	834	834
query45	193	176	177	176
query46	1183	750	736	736
query47	2021	1907	1899	1899
query48	518	393	404	393
query49	1084	432	411	411
query50	841	439	436	436
query51	7580	7353	7408	7353
query52	101	93	90	90
query53	256	181	181	181
query54	1319	478	475	475
query55	82	78	84	78
query56	280	252	259	252
query57	1326	1229	1219	1219
query58	244	220	240	220
query59	3246	2870	2951	2870
query60	287	271	286	271
query61	110	115	116	115
query62	859	724	689	689
query63	227	198	193	193
query64	4958	712	657	657
query65	3404	3365	3319	3319
query66	1516	313	312	312
query67	16132	15683	15404	15404
query68	4971	597	574	574
query69	431	268	270	268
query70	1194	1125	1121	1121
query71	337	260	250	250
query72	6216	4171	4020	4020
query73	759	353	363	353
query74	10603	9232	9237	9232
query75	3380	2655	2673	2655
query76	3019	1080	1154	1080
query77	378	281	268	268
query78	10580	9673	9511	9511
query79	2250	603	610	603
query80	1138	436	430	430
query81	548	220	220	220
query82	885	88	86	86
query83	238	143	139	139
query84	231	83	77	77
query85	1412	300	303	300
query86	450	292	302	292
query87	4389	4265	4230	4230
query88	4120	2367	2354	2354
query89	423	297	300	297
query90	2023	193	187	187
query91	187	149	152	149
query92	64	50	52	50
query93	2272	555	566	555
query94	951	295	314	295
query95	363	259	257	257
query96	639	275	278	275
query97	3300	3150	3158	3150
query98	218	202	204	202
query99	1492	1297	1304	1297
Total cold run time: 306400 ms
Total hot run time: 196881 ms

doris-robot · 2025-07-15T10:59:59Z

ClickBench: Total hot run time: 30.87 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d98e50753b2a23d4545cdaa3f2e9af1f5fe74be4, data reload: false
query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.11
query5	0.52	0.50	0.51
query6	1.13	0.73	0.73
query7	0.02	0.03	0.02
query8	0.04	0.03	0.03
query9	0.55	0.51	0.50
query10	0.55	0.57	0.57
query11	0.15	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	0.77	0.81	0.81
query15	0.86	0.83	0.82
query16	0.38	0.39	0.39
query17	1.05	1.07	1.05
query18	0.24	0.22	0.21
query19	1.90	1.76	1.80
query20	0.01	0.01	0.01
query21	15.39	0.63	0.59
query22	2.29	2.00	2.30
query23	16.99	1.06	0.86
query24	3.01	1.31	1.72
query25	0.23	0.11	0.21
query26	0.44	0.13	0.14
query27	0.05	0.04	0.04
query28	9.59	0.51	0.47
query29	12.61	3.24	3.27
query30	0.24	0.06	0.07
query31	2.86	0.38	0.38
query32	3.23	0.46	0.46
query33	2.98	3.03	3.03
query34	17.17	4.47	4.46
query35	4.54	4.52	4.54
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.62 s
Total hot run time: 30.87 s

dataroaring

LGTM

github-actions · 2025-07-15T11:37:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-15T11:37:04Z

PR approved by anyone and no changes requested.

deardeng requested a review from dataroaring as a code owner July 15, 2025 02:54

gavinchou reviewed Jul 15, 2025
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java Outdated

gavinchou reviewed Jul 15, 2025
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java Outdated

Fix
d98e507

dataroaring approved these changes Jul 15, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2025

github-actionsBot added the reviewed label Jul 15, 2025

dataroaring merged commit b6e3858 into apache:branch-3.0Jul 16, 2025

gavinchou mentioned this pull request Aug 15, 2025
Release Note 3.0.7 #54868
Closed

Navigation Menu

Uh oh!

branch-3.0: [fix](auto bucket)Fix auto bucket calc bucketnum err when partition size is invalid #52801 - #53250

branch-3.0: [fix](auto bucket)Fix auto bucket calc bucketnum err when partition size is invalid #52801#53250
dataroaring merged 2 commits into
apache:branch-3.0from
deardeng:pick_52801_to_doris_branch-3.0

deardeng commented Jul 15, 2025

Uh oh!

deardeng commented Jul 15, 2025

Uh oh!

Thearas commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

deardeng commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

dataroaring left a comment

Uh oh!

github-actionsBot commented Jul 15, 2025

Uh oh!

github-actionsBot commented Jul 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

deardeng commented Jul 15, 2025

Uh oh!

deardeng commented Jul 15, 2025

Uh oh!

Thearas commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

deardeng commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

doris-robot commented Jul 15, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 15, 2025

Uh oh!

github-actionsBot commented Jul 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants