[fix](mtmv) Fix track partition column fail when date_trunc in group by by seawinde · Pull Request #36175 · apache/doris

seawinde · 2024-06-12T07:45:41Z

Proposed changes

This is brought by #35562
At the pr above when you create partition materialized view as following, which would fail with the message Unable to find a suitable base table for partitioning:

 CREATE MATERIALIZED VIEW mvName
BUILD IMMEDIATE REFRESH AUTO ON MANUAL
partition by (date_trunc(month_alias, 'month'))
DISTRIBUTED BY RANDOM BUCKETS 2
PROPERTIES (
'replication_num' = '1'
)
AS
SELECT date_trunc(`k2`,'day') as month_alias, k3, count(*) FROM tableName group by date_trunc(`k2`,'day'), k3;

This pr supports to create partition materialized view when date_trunc in group by cluause.

doris-robot · 2024-06-12T07:45:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

seawinde · 2024-06-13T01:58:47Z

run buildall

doris-robot · 2024-06-13T02:42:23Z

TPC-H: Total hot run time: 39702 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9aac5f8880ef59e8ba8f1c1e3e68f2445bff86db, data reload: false
------ Round 1 ----------------------------------
q1	17626	4311	4284	4284
q2	2032	218	189	189
q3	10422	1155	1139	1139
q4	10212	842	761	761
q5	7460	2620	2619	2619
q6	220	139	136	136
q7	955	598	611	598
q8	9225	2040	2072	2040
q9	8807	6516	6460	6460
q10	8969	3760	3707	3707
q11	440	242	246	242
q12	428	231	222	222
q13	17764	2996	2967	2967
q14	267	211	215	211
q15	520	483	495	483
q16	515	378	370	370
q17	944	626	701	626
q18	8068	7533	7539	7533
q19	3726	1420	1486	1420
q20	640	313	303	303
q21	5007	3214	3057	3057
q22	384	335	336	335
Total cold run time: 114631 ms
Total hot run time: 39702 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4367	4228	4215	4215
q2	395	255	270	255
q3	2949	2744	2911	2744
q4	2040	1716	1703	1703
q5	5535	5577	5496	5496
q6	216	131	145	131
q7	2238	1837	1907	1837
q8	3281	3408	3408	3408
q9	8745	8628	8784	8628
q10	4059	3842	3885	3842
q11	610	513	473	473
q12	790	605	598	598
q13	17053	3216	3175	3175
q14	300	279	280	279
q15	535	478	476	476
q16	507	452	430	430
q17	1792	1486	1506	1486
q18	8053	7976	7840	7840
q19	1766	1525	1613	1525
q20	2067	1887	1912	1887
q21	5168	4923	4770	4770
q22	625	536	551	536
Total cold run time: 73091 ms
Total hot run time: 55734 ms

doris-robot · 2024-06-13T02:53:32Z

TPC-DS: Total hot run time: 174466 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9aac5f8880ef59e8ba8f1c1e3e68f2445bff86db, data reload: false
query1	940	389	374	374
query2	6460	2560	2353	2353
query3	6643	207	204	204
query4	19237	17430	17460	17430
query5	3596	495	496	495
query6	244	160	162	160
query7	4600	308	298	298
query8	331	310	293	293
query9	8500	2500	2474	2474
query10	567	290	289	289
query11	10718	10105	10192	10105
query12	118	89	85	85
query13	1645	364	365	364
query14	10146	7150	6447	6447
query15	234	188	185	185
query16	7683	273	267	267
query17	1423	536	517	517
query18	1956	300	276	276
query19	189	151	153	151
query20	91	81	82	81
query21	205	131	130	130
query22	4510	4126	4042	4042
query23	33603	33876	33525	33525
query24	10998	2930	2858	2858
query25	587	388	381	381
query26	707	161	150	150
query27	2309	347	346	346
query28	5674	2136	2133	2133
query29	884	669	630	630
query30	270	155	163	155
query31	974	802	758	758
query32	94	54	54	54
query33	694	291	315	291
query34	892	505	493	493
query35	765	663	621	621
query36	1128	975	934	934
query37	149	71	74	71
query38	2927	2814	2822	2814
query39	943	841	812	812
query40	210	130	130	130
query41	57	53	52	52
query42	115	102	106	102
query43	642	588	572	572
query44	1089	734	739	734
query45	197	158	168	158
query46	1085	747	729	729
query47	1854	1772	1748	1748
query48	382	310	303	303
query49	873	424	430	424
query50	779	399	413	399
query51	6810	6704	6701	6701
query52	103	102	95	95
query53	367	297	294	294
query54	891	458	454	454
query55	75	74	83	74
query56	303	274	281	274
query57	1156	1060	1085	1060
query58	255	257	270	257
query59	3775	3382	3262	3262
query60	315	298	300	298
query61	116	112	112	112
query62	636	482	440	440
query63	330	296	305	296
query64	8909	2324	1878	1878
query65	3243	3111	3094	3094
query66	772	338	337	337
query67	15530	15050	15106	15050
query68	6148	549	548	548
query69	728	485	480	480
query70	1186	1116	1198	1116
query71	499	271	281	271
query72	8323	5735	5527	5527
query73	784	343	329	329
query74	5869	5573	5482	5482
query75	4379	2642	2679	2642
query76	3833	1031	942	942
query77	752	304	305	304
query78	10246	9959	9759	9759
query79	5838	511	529	511
query80	1933	481	470	470
query81	578	220	229	220
query82	788	107	110	107
query83	268	167	167	167
query84	265	90	87	87
query85	1227	303	281	281
query86	459	280	296	280
query87	3275	3081	3089	3081
query88	4839	2485	2459	2459
query89	492	374	378	374
query90	1843	192	193	192
query91	136	100	99	99
query92	63	48	48	48
query93	5262	508	503	503
query94	1090	192	193	192
query95	416	323	320	320
query96	603	277	271	271
query97	3271	3046	3004	3004
query98	226	194	192	192
query99	1189	836	835	835
Total cold run time: 281311 ms
Total hot run time: 174466 ms

doris-robot · 2024-06-13T02:59:03Z

ClickBench: Total hot run time: 30.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9aac5f8880ef59e8ba8f1c1e3e68f2445bff86db, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.23	0.06	0.05
query4	1.64	0.08	0.07
query5	0.49	0.49	0.52
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.49	0.49
query10	0.55	0.56	0.53
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.78	0.81	0.78
query15	0.83	0.84	0.82
query16	0.37	0.37	0.36
query17	0.99	1.06	0.94
query18	0.22	0.27	0.23
query19	1.84	1.85	1.80
query20	0.02	0.01	0.01
query21	15.41	0.66	0.66
query22	4.80	6.64	1.83
query23	18.25	1.42	1.39
query24	2.11	0.24	0.21
query25	0.15	0.09	0.08
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.23	1.03	1.01
query29	12.62	3.29	3.24
query30	0.26	0.07	0.06
query31	2.87	0.38	0.38
query32	3.26	0.47	0.47
query33	2.88	2.92	2.92
query34	17.02	4.40	4.39
query35	4.47	4.48	4.53
query36	0.66	0.48	0.46
query37	0.19	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.04	0.03
query40	0.17	0.15	0.17
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.84 s
Total hot run time: 30.69 s

seawinde · 2024-06-19T13:45:51Z

run buildall

seawinde · 2024-06-20T01:19:21Z

run buildall

doris-robot · 2024-06-20T01:57:19Z

TPC-H: Total hot run time: 39415 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9963f381d91f4b1d57b351282f660a869270b16c, data reload: false
------ Round 1 ----------------------------------
q1	17629	4308	4253	4253
q2	2030	193	204	193
q3	10434	1083	1091	1083
q4	10188	744	825	744
q5	7477	2601	2615	2601
q6	220	133	131	131
q7	947	597	598	597
q8	9212	2037	2039	2037
q9	8759	6418	6416	6416
q10	8952	3699	3668	3668
q11	451	233	247	233
q12	472	232	219	219
q13	18936	2948	2956	2948
q14	271	222	230	222
q15	519	473	484	473
q16	536	371	369	369
q17	963	642	624	624
q18	7874	7463	7366	7366
q19	4937	1415	1407	1407
q20	644	322	328	322
q21	4882	3186	3230	3186
q22	385	327	323	323
Total cold run time: 116718 ms
Total hot run time: 39415 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4315	4252	4230	4230
q2	377	270	288	270
q3	2991	2800	2959	2800
q4	2010	1740	1726	1726
q5	5580	5493	5495	5493
q6	229	126	127	126
q7	2163	1746	1884	1746
q8	3252	3396	3371	3371
q9	8713	8673	8826	8673
q10	4052	3874	3814	3814
q11	585	478	466	466
q12	775	654	653	653
q13	15875	3142	3166	3142
q14	281	286	268	268
q15	531	485	484	484
q16	500	434	431	431
q17	1811	1493	1534	1493
q18	8065	7901	7793	7793
q19	1790	1668	1628	1628
q20	2140	1879	1882	1879
q21	5131	4966	5029	4966
q22	590	522	555	522
Total cold run time: 71756 ms
Total hot run time: 55974 ms

doris-robot · 2024-06-20T02:08:12Z

TPC-DS: Total hot run time: 172525 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9963f381d91f4b1d57b351282f660a869270b16c, data reload: false
query1	944	396	391	391
query2	6471	2319	2239	2239
query3	6633	206	211	206
query4	19482	17218	17117	17117
query5	3573	487	463	463
query6	243	175	164	164
query7	4591	296	285	285
query8	306	286	272	272
query9	8386	2401	2376	2376
query10	568	306	281	281
query11	10479	10002	10012	10002
query12	120	88	80	80
query13	1654	358	363	358
query14	9570	7812	7155	7155
query15	242	199	198	198
query16	7809	278	266	266
query17	1906	563	556	556
query18	1982	288	283	283
query19	206	162	157	157
query20	100	87	84	84
query21	205	134	132	132
query22	4625	4356	4305	4305
query23	34180	33521	33436	33436
query24	11001	2837	2780	2780
query25	612	354	346	346
query26	1147	147	156	147
query27	2288	319	317	317
query28	7121	2054	2031	2031
query29	879	600	598	598
query30	236	150	150	150
query31	916	719	718	718
query32	85	52	52	52
query33	735	275	289	275
query34	890	469	455	455
query35	741	644	617	617
query36	1092	933	921	921
query37	182	69	66	66
query38	2896	2855	2749	2749
query39	860	810	821	810
query40	200	123	120	120
query41	46	44	44	44
query42	121	96	99	96
query43	550	525	533	525
query44	1182	722	737	722
query45	186	159	161	159
query46	1072	717	696	696
query47	1847	1753	1774	1753
query48	370	305	290	290
query49	853	390	414	390
query50	750	389	380	380
query51	6797	6696	6604	6604
query52	104	90	94	90
query53	349	294	296	294
query54	903	435	439	435
query55	73	73	69	69
query56	272	251	269	251
query57	1111	1041	1026	1026
query58	279	242	263	242
query59	3231	3018	3238	3018
query60	290	258	268	258
query61	137	88	91	88
query62	609	459	431	431
query63	317	290	294	290
query64	8772	2230	1718	1718
query65	3170	3139	3091	3091
query66	728	323	325	323
query67	15357	14837	14843	14837
query68	4530	535	546	535
query69	561	382	395	382
query70	1151	1043	1056	1043
query71	408	262	277	262
query72	7101	5500	5300	5300
query73	749	315	316	315
query74	5882	5529	5563	5529
query75	3384	2655	2650	2650
query76	2580	887	956	887
query77	531	284	298	284
query78	10419	9764	9902	9764
query79	2795	512	518	512
query80	1184	470	484	470
query81	576	218	217	217
query82	1399	104	101	101
query83	260	167	166	166
query84	238	83	83	83
query85	1398	284	285	284
query86	441	299	287	287
query87	3291	3078	3077	3077
query88	4041	2334	2343	2334
query89	470	382	390	382
query90	1688	192	188	188
query91	131	101	158	101
query92	59	49	47	47
query93	2271	498	498	498
query94	988	186	181	181
query95	406	311	309	309
query96	586	265	269	265
query97	3227	3070	2982	2982
query98	220	197	190	190
query99	1166	847	822	822
Total cold run time: 269763 ms
Total hot run time: 172525 ms

doris-robot · 2024-06-20T02:13:44Z

ClickBench: Total hot run time: 31.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9963f381d91f4b1d57b351282f660a869270b16c, data reload: false
query1	0.04	0.04	0.03
query2	0.10	0.03	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.06
query5	0.50	0.49	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.54	0.48	0.50
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.77	0.78	0.77
query15	0.85	0.82	0.81
query16	0.37	0.37	0.36
query17	1.05	1.00	1.03
query18	0.23	0.25	0.22
query19	1.81	1.83	1.85
query20	0.01	0.02	0.01
query21	15.42	0.65	0.63
query22	4.56	6.93	2.22
query23	18.33	1.31	1.30
query24	2.12	0.22	0.23
query25	0.15	0.08	0.07
query26	0.27	0.17	0.18
query27	0.08	0.09	0.08
query28	13.19	1.02	1.01
query29	12.61	3.30	3.28
query30	0.26	0.07	0.05
query31	2.87	0.37	0.38
query32	3.30	0.46	0.47
query33	2.90	2.89	2.90
query34	17.06	4.49	4.52
query35	4.49	4.49	4.49
query36	0.66	0.46	0.47
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.86 s
Total hot run time: 31.05 s

zddr · 2024-06-20T03:51:19Z

+ 'replication_num' = '1'
+ )
+ AS
+ SELECT date_trunc(`k2`,'day') as month_alias, k1, count(*) FROM ${tableName} group by month_alias, k1;


It's best to call it day_alias

github-actions · 2024-06-20T03:53:04Z

PR approved by anyone and no changes requested.

github-actions · 2024-06-21T03:16:14Z

PR approved by at least one committer and no changes requested.

…by (#36175) This is brought by #35562 At the pr above when you create partition materialized view as following, which would fail with the message: Unable to find a suitable base table for partitioning CREATE MATERIALIZED VIEW mvName BUILD IMMEDIATE REFRESH AUTO ON MANUAL PARTITION BY (date_trunc(month_alias, 'month')) DISTRIBUTED BY RANDOM BUCKETS 2 PROPERTIES ( 'replication_num' = '1' ) AS SELECT date_trunc(`k2`,'day') AS month_alias, k3, count(*) FROM tableName GROUP BY date_trunc(`k2`,'day'), k3; This pr supports to create partition materialized view when `date_trunc` in group by cluause.

…by (apache#36175) This is brought by apache#35562 At the pr above when you create partition materialized view as following, which would fail with the message: Unable to find a suitable base table for partitioning CREATE MATERIALIZED VIEW mvName BUILD IMMEDIATE REFRESH AUTO ON MANUAL PARTITION BY (date_trunc(month_alias, 'month')) DISTRIBUTED BY RANDOM BUCKETS 2 PROPERTIES ( 'replication_num' = '1' ) AS SELECT date_trunc(`k2`,'day') AS month_alias, k3, count(*) FROM tableName GROUP BY date_trunc(`k2`,'day'), k3; This pr supports to create partition materialized view when `date_trunc` in group by cluause.

cherry-pick from master pr: #36318 commitId: c199947 pr: #36111 commitId: 35ebef6 pr: #36175 commitId: 4c8e66b pr: #36414 commitId: 5e009b5 pr: #36770 commitId: 19e2126 pr: #36567 commitId: 3da8351

…by (apache#36175) This is brought by apache#35562 At the pr above when you create partition materialized view as following, which would fail with the message: Unable to find a suitable base table for partitioning CREATE MATERIALIZED VIEW mvName BUILD IMMEDIATE REFRESH AUTO ON MANUAL PARTITION BY (date_trunc(month_alias, 'month')) DISTRIBUTED BY RANDOM BUCKETS 2 PROPERTIES ( 'replication_num' = '1' ) AS SELECT date_trunc(`k2`,'day') AS month_alias, k3, count(*) FROM tableName GROUP BY date_trunc(`k2`,'day'), k3; This pr supports to create partition materialized view when `date_trunc` in group by cluause.

[fix](mtmv) Fix track partition column fail when date_trunc in group by
9aac5f8

morrySnow added the dev/2.1.x label Jun 13, 2024

optimize the error info
9963f38

zddr approved these changes Jun 20, 2024
View reviewed changes

github-actionsBot added the reviewed label Jun 20, 2024

morrySnow approved these changes Jun 21, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 21, 2024

morrySnow merged commit 4c8e66b into apache:masterJun 21, 2024

seawinde mentioned this pull request Jul 11, 2024
[feature](mtmv) pick some mtmv pr from master #37651
Merged

morrySnow added dev/2.1.5-merged and removed dev/2.1.x labels Jul 12, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Navigation Menu

Uh oh!

[fix](mtmv) Fix track partition column fail when date_trunc in group by - #36175

[fix](mtmv) Fix track partition column fail when date_trunc in group by#36175
morrySnow merged 2 commits into
apache:masterfrom
seawinde:fix_date_trunc_not_expected

seawinde commented Jun 12, 2024

Uh oh!

doris-robot commented Jun 12, 2024

Uh oh!

seawinde commented Jun 13, 2024

Uh oh!

doris-robot commented Jun 13, 2024

Uh oh!

doris-robot commented Jun 13, 2024

Uh oh!

doris-robot commented Jun 13, 2024

Uh oh!

seawinde commented Jun 19, 2024

Uh oh!

seawinde commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

zddrJun 20, 2024

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

seawinde commented Jun 12, 2024

Proposed changes

Uh oh!

doris-robot commented Jun 12, 2024

Uh oh!

seawinde commented Jun 13, 2024

Uh oh!

doris-robot commented Jun 13, 2024

Uh oh!

doris-robot commented Jun 13, 2024

Uh oh!

doris-robot commented Jun 13, 2024

Uh oh!

seawinde commented Jun 19, 2024

Uh oh!

seawinde commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

zddrJun 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

github-actionsBot commented Jun 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants