[fix] (planner) support auto aggregation for random distributed table by DarvenDuan · Pull Request #33630 · apache/doris

DarvenDuan · 2024-04-14T05:35:16Z

Proposed changes

Issue Number: close #xxx

support auto aggregation for querying detail data of random distributed table: the same key column will return only one row.

step 1: fix legacy planner (√)
step 2: fix nereids planner (√)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-14T05:35:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

DarvenDuan · 2024-04-14T05:40:04Z

run buildall

doris-robot · 2024-04-14T06:18:39Z

TPC-H: Total hot run time: 38165 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3daf04bc6a177cca660cf2ba968ad577a3cd58da, data reload: false
------ Round 1 ----------------------------------
q1	17620	4263	4174	4174
q2	2010	188	180	180
q3	10472	1160	1148	1148
q4	10197	833	709	709
q5	7524	2678	2630	2630
q6	212	129	128	128
q7	976	599	583	583
q8	9214	2044	2038	2038
q9	7966	6575	6473	6473
q10	8579	3553	3480	3480
q11	457	239	228	228
q12	422	219	211	211
q13	18618	2924	2917	2917
q14	271	224	234	224
q15	523	481	480	480
q16	512	399	372	372
q17	958	732	745	732
q18	7320	6770	6684	6684
q19	5811	1538	1500	1500
q20	686	311	305	305
q21	3546	2661	2837	2661
q22	363	315	308	308
Total cold run time: 114257 ms
Total hot run time: 38165 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4439	4262	4254	4254
q2	371	271	267	267
q3	2977	2742	2773	2742
q4	1850	1555	1584	1555
q5	5359	5323	5325	5323
q6	209	122	121	121
q7	2247	1882	1911	1882
q8	3189	3312	3307	3307
q9	8594	8503	8657	8503
q10	4071	3921	3977	3921
q11	617	507	506	506
q12	794	623	650	623
q13	16820	3243	3156	3156
q14	317	276	319	276
q15	523	483	461	461
q16	511	462	444	444
q17	1830	1554	1487	1487
q18	8061	8208	7718	7718
q19	1680	1544	1590	1544
q20	2041	1872	1858	1858
q21	5136	4974	4843	4843
q22	575	504	473	473
Total cold run time: 72211 ms
Total hot run time: 55264 ms

doris-robot · 2024-04-14T06:30:27Z

TPC-DS: Total hot run time: 184601 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3daf04bc6a177cca660cf2ba968ad577a3cd58da, data reload: false
query1	1235	1112	1116	1112
query2	6194	2771	2507	2507
query3	6648	208	203	203
query4	36767	21653	21462	21462
query5	4159	388	399	388
query6	240	189	182	182
query7	4046	294	284	284
query8	223	171	167	167
query9	5770	2355	2292	2292
query10	361	239	241	239
query11	14578	14217	14136	14136
query12	141	90	83	83
query13	991	353	349	349
query14	9951	6838	6956	6838
query15	205	174	184	174
query16	6869	258	262	258
query17	1683	542	550	542
query18	1530	273	269	269
query19	185	149	151	149
query20	91	85	85	85
query21	198	129	121	121
query22	4988	4835	4804	4804
query23	33760	33118	33456	33118
query24	11198	2959	2984	2959
query25	531	410	372	372
query26	810	172	153	153
query27	3085	360	363	360
query28	6614	2118	2075	2075
query29	877	644	614	614
query30	311	182	167	167
query31	967	774	738	738
query32	59	54	51	51
query33	516	256	257	256
query34	931	500	493	493
query35	857	723	715	715
query36	1057	976	988	976
query37	116	73	76	73
query38	3728	3569	3651	3569
query39	1643	1577	1573	1573
query40	172	134	128	128
query41	46	44	46	44
query42	104	96	96	96
query43	606	547	578	547
query44	1356	727	713	713
query45	277	249	279	249
query46	1081	744	729	729
query47	2024	1957	1978	1957
query48	377	301	298	298
query49	832	373	357	357
query50	779	396	401	396
query51	6975	6858	6752	6752
query52	107	89	89	89
query53	341	277	279	277
query54	245	219	220	219
query55	73	70	70	70
query56	239	218	227	218
query57	1214	1143	1115	1115
query58	216	198	195	195
query59	3409	3381	3245	3245
query60	243	234	228	228
query61	92	87	102	87
query62	603	453	443	443
query63	304	283	283	283
query64	4147	3929	4197	3929
query65	3066	3029	3037	3029
query66	750	321	322	321
query67	15645	14889	14908	14889
query68	7022	550	543	543
query69	531	313	305	305
query70	1298	1190	1131	1131
query71	472	282	278	278
query72	6585	2732	2562	2562
query73	823	322	320	320
query74	7145	6410	6414	6410
query75	3145	2411	2360	2360
query76	4210	1135	1116	1116
query77	595	263	253	253
query78	10973	10162	10192	10162
query79	7357	522	529	522
query80	2113	442	445	442
query81	527	235	240	235
query82	1575	97	95	95
query83	338	174	169	169
query84	267	84	85	84
query85	1432	312	307	307
query86	463	297	278	278
query87	3788	3503	3528	3503
query88	6153	2263	2280	2263
query89	479	377	377	377
query90	1959	175	174	174
query91	121	95	96	95
query92	56	46	47	46
query93	6345	516	507	507
query94	1125	185	179	179
query95	381	289	290	289
query96	597	257	259	257
query97	2641	2490	2465	2465
query98	231	222	223	222
query99	1213	843	855	843
Total cold run time: 301396 ms
Total hot run time: 184601 ms

doris-robot · 2024-04-14T06:35:59Z

ClickBench: Total hot run time: 30.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3daf04bc6a177cca660cf2ba968ad577a3cd58da, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.09	0.10
query5	0.51	0.49	0.51
query6	1.43	0.64	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.50	0.51
query10	0.55	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.59	0.58
query14	0.76	0.77	0.77
query15	0.81	0.81	0.81
query16	0.37	0.36	0.38
query17	0.97	1.01	1.03
query18	0.22	0.24	0.22
query19	1.78	1.67	1.66
query20	0.01	0.01	0.01
query21	15.41	0.66	0.64
query22	4.37	7.44	1.90
query23	18.29	1.48	1.26
query24	2.13	0.22	0.21
query25	0.14	0.08	0.08
query26	0.27	0.16	0.15
query27	0.08	0.08	0.08
query28	13.35	1.00	0.99
query29	12.58	3.28	3.28
query30	0.27	0.05	0.06
query31	2.95	0.38	0.37
query32	3.22	0.47	0.46
query33	2.84	2.81	2.82
query34	17.28	4.34	4.43
query35	4.52	4.49	4.43
query36	0.65	0.46	0.46
query37	0.17	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.01 s
Total hot run time: 30.2 s

doris-robot · 2024-04-14T06:39:06Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3daf04bc6a177cca660cf2ba968ad577a3cd58da with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.3 seconds inserted 10000000 Rows, about 751K ops/s

DarvenDuan · 2024-04-14T07:49:16Z

run buildall

doris-robot · 2024-04-14T08:28:06Z

TPC-H: Total hot run time: 38061 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4f6fb1d9f0301fee11291ac546a958d6a2b3089, data reload: false
------ Round 1 ----------------------------------
q1	17609	4290	4225	4225
q2	2005	184	182	182
q3	10466	1149	1211	1149
q4	10191	726	829	726
q5	7542	2688	2641	2641
q6	219	132	130	130
q7	991	597	571	571
q8	9224	2044	2032	2032
q9	7903	6515	6504	6504
q10	8552	3536	3508	3508
q11	470	240	229	229
q12	472	220	205	205
q13	17783	2908	2929	2908
q14	272	220	233	220
q15	515	487	473	473
q16	502	373	377	373
q17	951	618	677	618
q18	7354	6662	6546	6546
q19	5423	1529	1486	1486
q20	692	314	308	308
q21	3467	2732	2836	2732
q22	364	295	304	295
Total cold run time: 112967 ms
Total hot run time: 38061 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4376	4277	4252	4252
q2	367	255	271	255
q3	3015	2756	2781	2756
q4	1861	1578	1557	1557
q5	5328	5316	5277	5277
q6	210	123	124	123
q7	2230	1879	1902	1879
q8	3186	3345	3318	3318
q9	8564	8533	8557	8533
q10	4093	3938	3920	3920
q11	641	521	510	510
q12	779	670	643	643
q13	17814	3270	3021	3021
q14	321	306	302	302
q15	519	480	472	472
q16	504	442	445	442
q17	1827	1530	1527	1527
q18	8113	7887	7856	7856
q19	1650	1554	1618	1554
q20	2038	1859	1816	1816
q21	5227	4892	4945	4892
q22	524	455	468	455
Total cold run time: 73187 ms
Total hot run time: 55360 ms

doris-robot · 2024-04-14T08:39:58Z

TPC-DS: Total hot run time: 183526 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4f6fb1d9f0301fee11291ac546a958d6a2b3089, data reload: false
query1	868	1117	1132	1117
query2	7448	2552	2323	2323
query3	6656	211	201	201
query4	37017	21437	21343	21343
query5	4151	390	388	388
query6	230	179	175	175
query7	4030	282	282	282
query8	212	170	172	170
query9	5771	2282	2259	2259
query10	361	232	241	232
query11	14803	14219	14275	14219
query12	135	93	86	86
query13	988	361	365	361
query14	8690	6907	6867	6867
query15	216	182	174	174
query16	7235	272	254	254
query17	1688	594	557	557
query18	1474	299	273	273
query19	210	156	154	154
query20	94	87	87	87
query21	198	131	125	125
query22	5045	4921	4851	4851
query23	33767	32849	33381	32849
query24	11203	2986	3002	2986
query25	568	420	404	404
query26	908	160	158	158
query27	3067	361	358	358
query28	6646	2078	2104	2078
query29	886	659	634	634
query30	284	177	173	173
query31	954	737	750	737
query32	63	120	54	54
query33	559	244	244	244
query34	892	485	505	485
query35	829	691	717	691
query36	1055	946	938	938
query37	108	70	73	70
query38	3666	3581	3539	3539
query39	1628	1568	1551	1551
query40	185	128	128	128
query41	47	44	43	43
query42	104	97	97	97
query43	579	541	541	541
query44	1316	734	723	723
query45	296	300	255	255
query46	1080	731	741	731
query47	2029	2006	1973	1973
query48	369	294	295	294
query49	834	379	364	364
query50	796	395	386	386
query51	6869	6810	6824	6810
query52	101	83	90	83
query53	340	282	276	276
query54	254	216	225	216
query55	72	70	69	69
query56	235	216	224	216
query57	1235	1132	1139	1132
query58	214	199	198	198
query59	3207	3348	3056	3056
query60	259	256	236	236
query61	111	90	86	86
query62	593	439	427	427
query63	307	273	271	271
query64	3961	4094	3974	3974
query65	3068	2998	3018	2998
query66	721	310	364	310
query67	15939	15033	14787	14787
query68	8820	543	550	543
query69	604	299	309	299
query70	1277	1206	1139	1139
query71	499	281	262	262
query72	6842	2644	2429	2429
query73	903	316	317	316
query74	7132	6367	6443	6367
query75	3512	2395	2317	2317
query76	5323	1156	1165	1156
query77	620	250	249	249
query78	10946	10372	10101	10101
query79	9868	520	514	514
query80	1983	458	420	420
query81	509	232	226	226
query82	818	91	91	91
query83	212	165	166	165
query84	258	84	79	79
query85	992	314	266	266
query86	420	293	315	293
query87	3739	3479	3492	3479
query88	5993	2264	2362	2264
query89	526	366	370	366
query90	1979	177	172	172
query91	119	96	98	96
query92	64	48	47	47
query93	6661	500	499	499
query94	1125	176	178	176
query95	378	286	285	285
query96	601	256	259	256
query97	2659	2448	2459	2448
query98	229	224	219	219
query99	1182	860	887	860
Total cold run time: 306517 ms
Total hot run time: 183526 ms

doris-robot · 2024-04-14T08:45:29Z

ClickBench: Total hot run time: 30.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a4f6fb1d9f0301fee11291ac546a958d6a2b3089, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.04	0.05
query4	1.68	0.06	0.07
query5	0.50	0.47	0.48
query6	1.44	0.64	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.50	0.49
query10	0.56	0.57	0.53
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	0.75	0.76	0.77
query15	0.82	0.80	0.80
query16	0.40	0.36	0.37
query17	1.01	1.01	1.00
query18	0.21	0.26	0.21
query19	1.84	1.80	1.67
query20	0.01	0.00	0.01
query21	15.40	0.64	0.64
query22	4.46	6.14	2.22
query23	18.33	1.38	1.24
query24	1.76	0.27	0.21
query25	0.14	0.08	0.08
query26	0.27	0.16	0.16
query27	0.07	0.08	0.07
query28	13.42	0.99	1.00
query29	12.61	3.29	3.28
query30	0.27	0.06	0.05
query31	2.88	0.37	0.37
query32	3.28	0.47	0.46
query33	2.84	2.76	2.82
query34	17.07	4.40	4.38
query35	4.46	4.46	4.43
query36	0.63	0.48	0.46
query37	0.19	0.16	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.18	0.15	0.14
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 109.75 s
Total hot run time: 30.35 s

doris-robot · 2024-04-14T08:48:35Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a4f6fb1d9f0301fee11291ac546a958d6a2b3089 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s

DarvenDuan · 2024-04-17T00:12:00Z

run p0

DarvenDuan · 2024-04-17T00:32:16Z

run buildall

doris-robot · 2024-04-17T01:08:17Z

TPC-H: Total hot run time: 38329 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2150a8a0e489dbc168529f4205f269619ccc98c, data reload: false
------ Round 1 ----------------------------------
q1	17637	4666	4207	4207
q2	2025	192	186	186
q3	10517	1189	1229	1189
q4	10191	786	706	706
q5	7497	2660	2668	2660
q6	220	132	131	131
q7	1012	614	588	588
q8	9229	2045	2035	2035
q9	7284	6575	6506	6506
q10	8578	3533	3531	3531
q11	435	229	233	229
q12	499	223	214	214
q13	17775	2954	2937	2937
q14	291	235	224	224
q15	518	493	480	480
q16	498	393	385	385
q17	948	611	625	611
q18	7300	6749	6799	6749
q19	7000	1538	1497	1497
q20	646	316	312	312
q21	3458	2651	2950	2651
q22	366	301	318	301
Total cold run time: 113924 ms
Total hot run time: 38329 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4345	4241	4222	4222
q2	362	275	276	275
q3	2983	2759	2764	2759
q4	1866	1591	1554	1554
q5	5309	5342	5280	5280
q6	209	124	122	122
q7	2251	1868	1850	1850
q8	3201	3340	3341	3340
q9	8547	8557	8669	8557
q10	4121	3913	4059	3913
q11	598	498	498	498
q12	791	635	632	632
q13	16439	3225	3176	3176
q14	327	306	275	275
q15	538	499	484	484
q16	490	446	459	446
q17	1813	1548	1507	1507
q18	8036	8131	7865	7865
q19	1668	1594	1535	1535
q20	2071	1893	1829	1829
q21	5089	5045	4980	4980
q22	548	466	462	462
Total cold run time: 71602 ms
Total hot run time: 55561 ms

doris-robot · 2024-04-17T01:19:58Z

TPC-DS: Total hot run time: 186263 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2150a8a0e489dbc168529f4205f269619ccc98c, data reload: false
query1	899	367	368	367
query2	6433	2579	2306	2306
query3	6650	205	210	205
query4	24222	21419	21385	21385
query5	4128	394	413	394
query6	278	189	173	173
query7	4586	294	290	290
query8	241	174	172	172
query9	8499	2366	2338	2338
query10	409	236	256	236
query11	14843	14254	14264	14254
query12	137	90	83	83
query13	1646	357	348	348
query14	9330	8016	7997	7997
query15	275	180	184	180
query16	8239	257	264	257
query17	1994	573	547	547
query18	2107	279	281	279
query19	323	157	145	145
query20	88	85	82	82
query21	197	129	124	124
query22	4978	4777	4843	4777
query23	33733	33094	33316	33094
query24	11058	3105	3114	3105
query25	583	382	389	382
query26	716	164	159	159
query27	2355	362	381	362
query28	6029	2098	2068	2068
query29	877	636	633	633
query30	300	179	179	179
query31	967	809	767	767
query32	90	53	54	53
query33	645	251	241	241
query34	886	494	500	494
query35	843	715	705	705
query36	1064	901	909	901
query37	119	74	72	72
query38	3449	3369	3346	3346
query39	1617	1593	1719	1593
query40	177	129	129	129
query41	54	42	43	42
query42	106	97	95	95
query43	592	540	529	529
query44	1120	746	741	741
query45	285	280	242	242
query46	1095	774	736	736
query47	2046	1913	1947	1913
query48	382	299	309	299
query49	834	379	391	379
query50	807	397	390	390
query51	6881	6756	6797	6756
query52	97	96	88	88
query53	339	273	271	271
query54	298	236	244	236
query55	74	68	69	68
query56	237	222	219	219
query57	1204	1142	1138	1138
query58	216	194	199	194
query59	3352	3129	3168	3129
query60	251	236	231	231
query61	89	87	88	87
query62	610	454	444	444
query63	304	281	278	278
query64	4703	4179	3981	3981
query65	3037	3046	3076	3046
query66	755	344	339	339
query67	15177	14998	15020	14998
query68	5196	547	548	547
query69	514	307	303	303
query70	1201	1173	1215	1173
query71	1420	1281	1271	1271
query72	6499	2745	2547	2547
query73	725	323	321	321
query74	6747	6470	6380	6380
query75	3371	2672	2597	2597
query76	3418	991	903	903
query77	470	267	269	267
query78	10793	10206	10136	10136
query79	8512	527	536	527
query80	2378	447	475	447
query81	521	243	256	243
query82	1360	102	95	95
query83	314	166	166	166
query84	266	85	85	85
query85	1788	265	262	262
query86	478	298	292	292
query87	3474	3279	3274	3274
query88	5313	2416	2425	2416
query89	466	366	368	366
query90	1943	181	182	181
query91	126	98	98	98
query92	61	50	53	50
query93	6295	514	506	506
query94	1086	181	178	178
query95	382	301	291	291
query96	610	276	267	267
query97	3105	2918	2950	2918
query98	232	226	217	217
query99	1231	848	853	848
Total cold run time: 291250 ms
Total hot run time: 186263 ms

doris-robot · 2024-04-17T01:25:29Z

ClickBench: Total hot run time: 30.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2150a8a0e489dbc168529f4205f269619ccc98c, data reload: false
query1	0.03	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.07
query5	0.50	0.49	0.50
query6	1.47	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.47	0.49
query10	0.54	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.62	0.58
query14	0.75	0.78	0.77
query15	0.84	0.81	0.81
query16	0.36	0.37	0.35
query17	0.95	0.95	1.02
query18	0.19	0.26	0.24
query19	1.75	1.65	1.65
query20	0.02	0.01	0.01
query21	15.40	0.65	0.64
query22	4.32	7.14	1.85
query23	18.31	1.45	1.32
query24	1.80	0.27	0.20
query25	0.15	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.33	1.00	0.98
query29	12.56	3.26	3.27
query30	0.26	0.07	0.06
query31	2.85	0.38	0.38
query32	3.30	0.46	0.48
query33	2.77	2.86	2.84
query34	17.14	4.44	4.51
query35	4.52	4.51	4.47
query36	0.64	0.49	0.46
query37	0.19	0.15	0.16
query38	0.14	0.14	0.15
query39	0.05	0.04	0.05
query40	0.18	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.03	0.03	0.04
Total cold run time: 109.31 s
Total hot run time: 30.35 s

DarvenDuan · 2024-04-17T01:26:33Z

run p0

doris-robot · 2024-04-17T01:28:35Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d2150a8a0e489dbc168529f4205f269619ccc98c with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s

starocean999 · 2024-05-13T03:34:03Z

Generally, we can only fix the bug for nereids planner, unless there is a strong needs for branch 2.0.(the old planner will be discarded very soon). And we should consider not only MAX, MIN, SUM and also other agg types like HLL_UNION, BITMAP_UNION, QUANTILE_UNION and GENERIC(agg_state related)

starocean999 · 2024-05-16T07:48:12Z

run buildall

DarvenDuan · 2024-05-16T08:49:34Z

run p0

starocean999

LGTM

github-actions · 2024-05-16T10:48:20Z

PR approved by at least one committer and no changes requested.

…ed table (#33630) support auto aggregation for querying detail data of random distributed table: the same key column will return only one row.

## Proposed changes Bug description: In PR #33630, Doris supports auto aggregation for random distributed table, but it not only effects query statements, so if we delete from a random distributed table, will get an error because of unexpectedly rewriting. ``` CREATE TABLE `test_tbl` ( `k` INT NULL, `v` BIGINT SUM NULL ) ENGINE=OLAP AGGREGATE KEY(`k`) DISTRIBUTED BY RANDOM BUCKETS AUTO; mysql > delete from test_tbl where k=1; ERROR 1105 (HY000): errCode = 2, detailMessage = Where clause only supports compound predicate, binary predicate, is_null predicate or in predicate. ``` fix: Check whether it is a query statement before rewriting.

…ed table (apache#33630) support auto aggregation for querying detail data of random distributed table: the same key column will return only one row.

## Proposed changes Bug description: In PR apache#33630, Doris supports auto aggregation for random distributed table, but it not only effects query statements, so if we delete from a random distributed table, will get an error because of unexpectedly rewriting. ``` CREATE TABLE `test_tbl` ( `k` INT NULL, `v` BIGINT SUM NULL ) ENGINE=OLAP AGGREGATE KEY(`k`) DISTRIBUTED BY RANDOM BUCKETS AUTO; mysql > delete from test_tbl where k=1; ERROR 1105 (HY000): errCode = 2, detailMessage = Where clause only supports compound predicate, binary predicate, is_null predicate or in predicate. ``` fix: Check whether it is a query statement before rewriting.

morrySnow reviewed Apr 15, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java

starocean999 reviewed Apr 15, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java Outdated

starocean999 reviewed Apr 15, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java Outdated

DarvenDuan force-pushed the optimize_random_distribute branch from 26c6eeb to e005f3eCompare April 15, 2024 10:13

DarvenDuan marked this pull request as draft April 15, 2024 11:04

DarvenDuan force-pushed the optimize_random_distribute branch 4 times, most recently from c932047 to d2150a8Compare April 16, 2024 15:32

DarvenDuan marked this pull request as ready for review April 16, 2024 15:57

DarvenDuan requested review from morrySnow and starocean999 April 17, 2024 00:32

DarvenDuan added 6 commits May 16, 2024 14:15

support merge on read for random distributed table on legacy planner
11b7999

construct an agg node for pre-agg when querying random distributed table
0e07454

optimize for agg function
ed3e13a

support merge on read for random distributed table on nereids planner
ff21533

fix group by value column
26294cd

fix count distinct value column
ad5fbd1

DarvenDuan force-pushed the optimize_random_distribute branch from da53339 to 3d41b98Compare May 16, 2024 06:20

DarvenDuan marked this pull request as ready for review May 16, 2024 06:26

DarvenDuan requested a review from starocean999 May 16, 2024 06:26

support more aggregateType
d4a168d

DarvenDuan force-pushed the optimize_random_distribute branch from 3d41b98 to d4a168dCompare May 16, 2024 07:17

starocean999 approved these changes May 16, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 16, 2024

morrySnow approved these changes May 17, 2024
View reviewed changes

morrySnow merged commit d23b409 into apache:masterMay 17, 2024

DarvenDuan deleted the optimize_random_distribute branch May 17, 2024 09:31

morrySnow added the dev/2.1.4-merged label May 23, 2024

cambyzju mentioned this pull request Jun 13, 2024
[fix](plan) wrong result for random distributed agg table with all keys not null #36270
Merged

DarvenDuan mentioned this pull request Jul 17, 2024
[bug](delete) fix delete random distributed tbl #37985
Merged

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Uh oh!

Conversation

DarvenDuan commented Apr 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

DarvenDuan commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

DarvenDuan commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

doris-robot commented Apr 14, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarvenDuan commented Apr 17, 2024

Uh oh!

DarvenDuan commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

DarvenDuan commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

starocean999 commented May 13, 2024

Uh oh!

starocean999 commented May 16, 2024

Uh oh!

DarvenDuan commented May 16, 2024

Uh oh!

starocean999 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DarvenDuan commented Apr 14, 2024•
edited
Loading