Skip to content

[fix](mtmv) Fix partition mv rewrite result wrong - #35236

Merged
morrySnow merged 6 commits into
apache:masterfrom
seawinde:fix_partition_mv_rewrite_result_wrong
May 29, 2024
Merged

[fix](mtmv) Fix partition mv rewrite result wrong#35236
morrySnow merged 6 commits into
apache:masterfrom
seawinde:fix_partition_mv_rewrite_result_wrong

Conversation

@seawinde

@seawindeseawinde commented May 22, 2024

Copy link
Copy Markdown
Member

Proposed changes

this is brought by #33800
if mv is partitioned materialzied view,
the data will be wrong by using the hited materialized view when the paritions in related base partiton table are deleted, created and so on.
this fix the problem.

if SET enable_materialized_view_union_rewrite=true; this will use the materializd view and make sure the data is corrent
if SET enable_materialized_view_union_rewrite=false; this will query base table directly to make sure the data is right

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented May 22, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde

seawinde commented May 22, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented May 22, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41096 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f8062183469ea5d1e80edc918a0ec33f8aa628f, data reload: false
------ Round 1 ----------------------------------
q1	17628	4428	4298	4298
q2	2011	195	199	195
q3	10486	1254	1209	1209
q4	10649	786	791	786
q5	7546	2727	2747	2727
q6	249	135	137	135
q7	982	637	634	634
q8	9775	2118	2115	2115
q9	9173	6675	6690	6675
q10	9218	3804	3739	3739
q11	459	245	233	233
q12	514	216	222	216
q13	17769	2994	2984	2984
q14	251	222	223	222
q15	518	464	486	464
q16	527	392	387	387
q17	963	636	672	636
q18	8168	7505	7467	7467
q19	6634	1565	1511	1511
q20	664	319	324	319
q21	5008	3860	3898	3860
q22	348	293	284	284
Total cold run time: 119540 ms
Total hot run time: 41096 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4339	4256	4206	4206
q2	379	283	271	271
q3	2992	2781	2708	2708
q4	1875	1605	1611	1605
q5	5307	5316	5309	5309
q6	211	126	128	126
q7	2115	1706	1779	1706
q8	3228	3352	3324	3324
q9	8418	8384	8369	8369
q10	3858	3675	3709	3675
q11	578	494	484	484
q12	761	632	605	605
q13	16575	2981	3008	2981
q14	288	262	283	262
q15	511	488	475	475
q16	466	413	435	413
q17	1764	1481	1472	1472
q18	7692	7729	7471	7471
q19	1672	1587	1571	1571
q20	1965	1768	1832	1768
q21	9561	4741	4608	4608
q22	597	502	505	502
Total cold run time: 75152 ms
Total hot run time: 53911 ms

@seawinde

seawinde commented May 22, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented May 22, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41816 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b72023e9266d6b3db9c7485861aec8cd6b9af0e, data reload: false
------ Round 1 ----------------------------------
q1	17610	4315	4443	4315
q2	2013	197	198	197
q3	10440	1235	1235	1235
q4	10194	889	739	739
q5	7474	2710	2620	2620
q6	234	138	143	138
q7	970	616	608	608
q8	9239	2125	2094	2094
q9	9875	6696	6648	6648
q10	9332	3884	3874	3874
q11	444	241	239	239
q12	458	231	232	231
q13	17253	3179	3240	3179
q14	266	234	233	233
q15	514	465	469	465
q16	487	382	384	382
q17	981	723	714	714
q18	8433	7809	7860	7809
q19	4561	1530	1506	1506
q20	634	321	318	318
q21	5176	4039	3995	3995
q22	343	283	277	277
Total cold run time: 116931 ms
Total hot run time: 41816 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4588	4415	4395	4395
q2	376	271	276	271
q3	3131	2893	3006	2893
q4	1869	1600	1581	1581
q5	5476	5523	5460	5460
q6	219	131	125	125
q7	2170	1755	1841	1755
q8	3301	3365	3375	3365
q9	8550	8674	8682	8674
q10	4025	3709	3806	3709
q11	584	494	488	488
q12	805	638	602	602
q13	16887	3141	3137	3137
q14	310	274	260	260
q15	525	493	485	485
q16	483	450	434	434
q17	1801	1541	1490	1490
q18	7750	7521	7445	7445
q19	1646	1572	1542	1542
q20	1992	1800	1789	1789
q21	10506	4788	4695	4695
q22	567	494	479	479
Total cold run time: 77561 ms
Total hot run time: 55074 ms

@doris-robot

doris-robot commented May 22, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 172210 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b72023e9266d6b3db9c7485861aec8cd6b9af0e, data reload: false
query1	916	373	366	366
query2	6439	2420	2478	2420
query3	6647	210	214	210
query4	23190	17423	17390	17390
query5	4102	423	418	418
query6	243	155	152	152
query7	4584	306	301	301
query8	244	182	172	172
query9	8532	2387	2364	2364
query10	468	290	272	272
query11	10705	10137	9998	9998
query12	134	92	91	91
query13	1641	398	366	366
query14	9199	7551	6736	6736
query15	217	171	175	171
query16	7895	269	269	269
query17	1703	538	535	535
query18	2027	266	266	266
query19	201	162	156	156
query20	98	84	86	84
query21	188	129	127	127
query22	4328	3875	3890	3875
query23	33505	33164	32965	32965
query24	10575	2796	2929	2796
query25	557	355	366	355
query26	707	174	153	153
query27	2210	323	336	323
query28	5595	2047	2068	2047
query29	847	608	603	603
query30	231	169	173	169
query31	989	773	746	746
query32	99	52	55	52
query33	656	295	258	258
query34	889	473	469	469
query35	748	596	572	572
query36	1108	910	903	903
query37	110	70	72	70
query38	2914	2820	2727	2727
query39	872	790	813	790
query40	194	154	124	124
query41	45	43	47	43
query42	107	99	97	97
query43	603	568	541	541
query44	1108	736	733	733
query45	185	171	161	161
query46	1064	727	722	722
query47	1869	1781	1776	1776
query48	371	300	295	295
query49	857	387	393	387
query50	779	398	398	398
query51	6984	6880	6823	6823
query52	111	97	94	94
query53	351	288	289	288
query54	873	431	430	430
query55	78	79	73	73
query56	268	244	272	244
query57	1127	1035	1052	1035
query58	257	223	210	210
query59	3405	3078	3272	3078
query60	284	255	265	255
query61	90	88	92	88
query62	615	453	457	453
query63	313	292	294	292
query64	8526	2246	1704	1704
query65	3153	3093	3099	3093
query66	782	322	321	321
query67	15085	14714	14903	14714
query68	5576	562	566	562
query69	498	275	268	268
query70	1151	1080	1153	1080
query71	433	275	277	275
query72	8560	5561	5485	5485
query73	788	340	329	329
query74	6115	5697	5614	5614
query75	3527	2608	2643	2608
query76	3297	998	1014	998
query77	576	278	277	277
query78	10233	9913	9738	9738
query79	2087	517	528	517
query80	1358	471	466	466
query81	524	251	241	241
query82	1281	103	98	98
query83	243	190	184	184
query84	266	91	93	91
query85	1435	331	311	311
query86	454	302	284	284
query87	3350	3096	3137	3096
query88	3791	2375	2375	2375
query89	490	392	381	381
query90	1995	203	197	197
query91	140	110	110	110
query92	61	50	51	50
query93	2071	514	507	507
query94	1238	201	204	201
query95	434	330	336	330
query96	596	268	270	268
query97	3204	3001	3002	3001
query98	246	232	215	215
query99	1228	844	851	844
Total cold run time: 273043 ms
Total hot run time: 172210 ms

@doris-robot

doris-robot commented May 22, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b72023e9266d6b3db9c7485861aec8cd6b9af0e, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.49	0.51
query6	1.12	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.52	0.49	0.50
query10	0.55	0.55	0.53
query11	0.16	0.12	0.11
query12	0.14	0.11	0.12
query13	0.59	0.59	0.59
query14	0.77	0.77	0.77
query15	0.84	0.80	0.83
query16	0.35	0.37	0.38
query17	1.01	0.98	0.94
query18	0.21	0.25	0.22
query19	1.79	1.69	1.71
query20	0.02	0.01	0.01
query21	15.41	0.74	0.69
query22	4.09	7.19	2.56
query23	18.28	1.44	1.28
query24	1.40	0.42	0.22
query25	0.14	0.09	0.09
query26	0.28	0.16	0.16
query27	0.07	0.09	0.08
query28	13.29	1.02	1.00
query29	12.96	3.34	3.25
query30	0.25	0.06	0.05
query31	2.88	0.37	0.39
query32	3.30	0.47	0.46
query33	2.82	2.87	2.82
query34	17.18	4.45	4.40
query35	4.53	4.47	4.59
query36	0.68	0.48	0.45
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 108.97 s
Total hot run time: 30.99 s

@seawinde
seawindeforce-pushed the fix_partition_mv_rewrite_result_wrong branch 2 times, most recently from b252757 to cfb4a43CompareMay 23, 2024 11:56
@seawinde

seawinde commented May 23, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented May 24, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40832 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 395b20f9d9bf83aa459ee6951f0d37e709061002, data reload: false
------ Round 1 ----------------------------------
q1	17612	4225	4198	4198
q2	2012	186	186	186
q3	10470	1237	1171	1171
q4	10197	745	843	745
q5	7480	2701	2801	2701
q6	219	134	137	134
q7	965	604	602	602
q8	9219	2129	2094	2094
q9	9408	6732	6665	6665
q10	9365	3991	3917	3917
q11	447	249	234	234
q12	514	252	219	219
q13	17385	3177	3227	3177
q14	273	215	210	210
q15	505	471	480	471
q16	486	382	397	382
q17	972	648	686	648
q18	8385	7845	7806	7806
q19	4497	1533	1474	1474
q20	636	320	312	312
q21	5208	3212	3257	3212
q22	366	274	279	274
Total cold run time: 116621 ms
Total hot run time: 40832 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4450	4449	4448	4448
q2	379	268	276	268
q3	3186	2968	2921	2921
q4	1998	1605	1604	1604
q5	5329	5468	5480	5468
q6	222	124	127	124
q7	2229	1797	1803	1797
q8	3260	3452	3380	3380
q9	8578	8636	8667	8636
q10	4091	3745	3733	3733
q11	591	492	497	492
q12	795	605	612	605
q13	16919	3147	3165	3147
q14	297	258	270	258
q15	523	496	488	488
q16	494	449	451	449
q17	1820	1525	1475	1475
q18	7621	7715	7480	7480
q19	1664	1550	1555	1550
q20	2017	1797	1776	1776
q21	12018	4605	4639	4605
q22	553	479	471	471
Total cold run time: 79034 ms
Total hot run time: 55175 ms

@doris-robot

doris-robot commented May 24, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 168496 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 395b20f9d9bf83aa459ee6951f0d37e709061002, data reload: false
query1	922	392	373	373
query2	6437	2402	2205	2205
query3	6635	210	210	210
query4	19818	17212	17322	17212
query5	4118	423	450	423
query6	235	164	155	155
query7	4594	305	306	305
query8	236	184	179	179
query9	8726	2360	2342	2342
query10	446	281	260	260
query11	10391	10105	9989	9989
query12	141	89	92	89
query13	1642	370	366	366
query14	8965	6958	6884	6884
query15	209	170	169	169
query16	7439	265	263	263
query17	1292	525	501	501
query18	1910	275	270	270
query19	191	147	151	147
query20	91	84	87	84
query21	198	125	131	125
query22	4092	3998	3826	3826
query23	33807	33050	33127	33050
query24	6944	2854	2841	2841
query25	550	358	358	358
query26	707	159	161	159
query27	1949	331	331	331
query28	3749	2028	2029	2028
query29	846	609	623	609
query30	281	176	172	172
query31	959	770	736	736
query32	96	53	54	53
query33	502	262	264	262
query34	860	463	464	463
query35	726	599	597	597
query36	1053	911	908	908
query37	108	69	73	69
query38	2883	2765	2771	2765
query39	863	809	780	780
query40	197	127	125	125
query41	48	44	43	43
query42	104	95	98	95
query43	591	542	545	542
query44	1087	715	734	715
query45	186	160	159	159
query46	1069	736	724	724
query47	1828	1755	1754	1754
query48	386	293	301	293
query49	768	377	395	377
query50	781	384	392	384
query51	6853	6761	6859	6761
query52	104	87	93	87
query53	354	285	291	285
query54	528	424	424	424
query55	73	78	74	74
query56	276	254	248	248
query57	1134	1035	1062	1035
query58	233	217	209	209
query59	3364	3118	3137	3118
query60	276	253	248	248
query61	90	90	86	86
query62	570	473	452	452
query63	320	289	291	289
query64	8477	2239	1697	1697
query65	3137	3117	3114	3114
query66	834	336	319	319
query67	15067	14698	15063	14698
query68	4525	540	547	540
query69	433	269	315	269
query70	1155	1064	1159	1064
query71	398	283	262	262
query72	7677	2695	2590	2590
query73	709	325	320	320
query74	6097	5504	5581	5504
query75	3317	2625	2645	2625
query76	2270	973	949	949
query77	374	269	268	268
query78	10363	9631	9679	9631
query79	2235	532	522	522
query80	815	454	440	440
query81	514	243	246	243
query82	1087	97	94	94
query83	236	169	172	169
query84	251	85	82	82
query85	1152	275	314	275
query86	463	327	336	327
query87	3265	3132	3036	3036
query88	4223	2317	2322	2317
query89	471	404	398	398
query90	2106	189	190	189
query91	125	96	95	95
query92	60	49	51	49
query93	2075	514	501	501
query94	1123	190	183	183
query95	396	308	310	308
query96	592	271	270	270
query97	3177	2998	2996	2996
query98	251	224	221	221
query99	1151	857	846	846
Total cold run time: 257432 ms
Total hot run time: 168496 ms

@doris-robot

doris-robot commented May 24, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 395b20f9d9bf83aa459ee6951f0d37e709061002, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.48	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.05
query9	0.55	0.49	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.13	0.12	0.12
query13	0.59	0.59	0.59
query14	0.78	0.77	0.78
query15	0.82	0.82	0.80
query16	0.35	0.38	0.35
query17	0.99	0.98	1.00
query18	0.20	0.25	0.23
query19	1.76	1.69	1.65
query20	0.01	0.00	0.01
query21	15.56	0.68	0.66
query22	4.27	7.77	1.73
query23	18.26	1.41	1.27
query24	1.59	0.23	0.30
query25	0.15	0.09	0.08
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.35	1.01	0.98
query29	13.84	3.27	3.22
query30	0.23	0.06	0.05
query31	2.88	0.38	0.39
query32	3.26	0.47	0.46
query33	2.89	2.91	2.90
query34	17.29	4.48	4.46
query35	4.54	4.61	4.66
query36	0.64	0.45	0.46
query37	0.17	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.06	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.4 s
Total hot run time: 30.42 s

morrySnow
morrySnow reviewed May 27, 2024
View reviewed changes
Comment threadregression-test/suites/nereids_rules_p0/mv/partition_mv_rewrite.groovy

sleep(5000)
sql "SET enable_materialized_view_rewrite=false"
order_qt_query_5_0_before "${all_partition_sql}"

@morrySnowmorrySnowMay 24, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use meaningful case name

Comment threadregression-test/suites/nereids_rules_p0/mv/partition_mv_rewrite.groovy Outdated
(1, 2, 3, 4, 5.5, 6.5, 7.5, 8.5, 'o', 'k', '2023-10-21', '2023-10-21', '2023-10-21', 'a', 'b', 'yyyyyyyyy');
"""

sleep(5000)

@morrySnowmorrySnowMay 24, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not use sleep, real check what u want to wait

@seawinde
seawindeforce-pushed the fix_partition_mv_rewrite_result_wrong branch from 4bc1130 to aeb9702CompareMay 27, 2024 09:21
@seawinde

seawinde commented May 27, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

1 similar comment
@seawinde

seawinde commented May 27, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41790 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8867d3faf23851b9781409a407d7f0d0175cf7f4, data reload: false
------ Round 1 ----------------------------------
q1	17615	4340	4272	4272
q2	2013	181	197	181
q3	10480	1255	1271	1255
q4	10179	795	764	764
q5	7490	2731	2624	2624
q6	223	132	136	132
q7	978	607	621	607
q8	9207	2137	2114	2114
q9	9248	6760	6854	6760
q10	9284	3883	3896	3883
q11	441	251	241	241
q12	545	212	212	212
q13	17300	3193	3240	3193
q14	245	217	215	215
q15	518	462	465	462
q16	514	392	399	392
q17	969	698	643	643
q18	8376	7911	7732	7732
q19	4275	1554	1559	1554
q20	642	313	309	309
q21	5301	3964	3977	3964
q22	346	288	281	281
Total cold run time: 116189 ms
Total hot run time: 41790 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4543	4436	4346	4346
q2	378	280	275	275
q3	3194	2932	2911	2911
q4	1881	1640	1612	1612
q5	5584	5495	5488	5488
q6	209	123	123	123
q7	2243	1866	1794	1794
q8	3254	3380	3356	3356
q9	8668	8710	8636	8636
q10	3924	3871	3865	3865
q11	619	487	497	487
q12	802	640	625	625
q13	15908	3129	3129	3129
q14	300	273	260	260
q15	516	502	490	490
q16	500	413	428	413
q17	1770	1481	1483	1481
q18	7683	7685	7529	7529
q19	1704	1551	1562	1551
q20	2046	1773	1787	1773
q21	9911	4844	4750	4750
q22	544	486	463	463
Total cold run time: 76181 ms
Total hot run time: 55357 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 171596 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8867d3faf23851b9781409a407d7f0d0175cf7f4, data reload: false
query1	928	387	373	373
query2	6440	2394	2578	2394
query3	6644	212	206	206
query4	19558	17383	17296	17296
query5	4148	408	413	408
query6	245	157	167	157
query7	4586	293	285	285
query8	236	185	184	184
query9	8663	2324	2297	2297
query10	455	280	282	280
query11	10607	10136	10114	10114
query12	138	93	90	90
query13	1647	367	366	366
query14	9801	6574	8158	6574
query15	218	173	163	163
query16	7719	266	268	266
query17	1711	514	493	493
query18	1868	280	270	270
query19	197	152	151	151
query20	90	84	90	84
query21	189	128	133	128
query22	4181	4035	3810	3810
query23	33586	33091	33229	33091
query24	6652	2865	2852	2852
query25	462	352	354	352
query26	694	152	154	152
query27	1889	317	309	309
query28	3872	2051	2021	2021
query29	820	592	613	592
query30	232	148	152	148
query31	910	768	749	749
query32	58	52	51	51
query33	514	259	258	258
query34	852	466	472	466
query35	709	598	582	582
query36	1027	925	888	888
query37	101	64	65	64
query38	2862	2789	2788	2788
query39	861	786	790	786
query40	205	124	122	122
query41	48	44	42	42
query42	103	93	97	93
query43	567	573	557	557
query44	1094	729	742	729
query45	173	165	167	165
query46	1046	716	706	706
query47	1902	1800	1825	1800
query48	369	291	304	291
query49	753	389	414	389
query50	788	384	377	377
query51	6835	6701	6816	6701
query52	97	94	87	87
query53	348	287	280	280
query54	516	427	425	425
query55	73	73	72	72
query56	256	241	244	241
query57	1163	1042	1024	1024
query58	229	210	211	210
query59	3401	3358	3157	3157
query60	274	271	259	259
query61	95	91	91	91
query62	543	464	450	450
query63	306	282	282	282
query64	2542	1752	1696	1696
query65	3198	3078	3106	3078
query66	780	334	322	322
query67	15584	14808	14991	14808
query68	4543	518	511	511
query69	438	266	285	266
query70	1127	1085	1153	1085
query71	372	272	269	269
query72	7888	5099	5758	5099
query73	726	326	320	320
query74	6079	5700	5580	5580
query75	3297	2624	2636	2624
query76	2313	992	976	976
query77	395	267	266	266
query78	10703	9850	9870	9850
query79	2473	505	515	505
query80	1076	454	445	445
query81	516	228	220	220
query82	702	92	90	90
query83	257	177	172	172
query84	255	89	95	89
query85	1042	325	316	316
query86	422	296	315	296
query87	3307	3146	3151	3146
query88	3974	2413	2443	2413
query89	484	394	376	376
query90	2082	192	184	184
query91	130	110	106	106
query92	72	50	49	49
query93	2102	481	483	481
query94	1099	198	188	188
query95	420	315	319	315
query96	589	269	263	263
query97	3219	3032	3070	3032
query98	239	220	221	220
query99	1232	876	852	852
Total cold run time: 253231 ms
Total hot run time: 171596 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8867d3faf23851b9781409a407d7f0d0175cf7f4, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.69	0.07	0.07
query5	0.51	0.52	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.54	0.50	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.58	0.59	0.59
query14	0.76	0.78	0.78
query15	0.82	0.81	0.82
query16	0.34	0.37	0.37
query17	0.93	0.98	0.96
query18	0.21	0.24	0.24
query19	1.86	1.72	1.66
query20	0.02	0.01	0.01
query21	15.49	0.68	0.67
query22	4.34	6.06	2.18
query23	18.31	1.27	1.25
query24	1.39	0.28	0.33
query25	0.16	0.07	0.08
query26	0.26	0.16	0.16
query27	0.08	0.07	0.07
query28	13.29	1.00	0.98
query29	13.26	3.29	3.30
query30	0.24	0.06	0.05
query31	2.90	0.39	0.38
query32	3.29	0.46	0.47
query33	2.87	2.86	2.84
query34	17.12	4.43	4.47
query35	4.50	4.50	4.57
query36	0.66	0.47	0.46
query37	0.17	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.15
query41	0.10	0.04	0.04
query42	0.06	0.05	0.04
query43	0.05	0.03	0.04
Total cold run time: 109.52 s
Total hot run time: 30.76 s

@morrySnow

morrySnow commented May 27, 2024

Copy link
Copy Markdown
Contributor

run feut

@seawinde
seawindeforce-pushed the fix_partition_mv_rewrite_result_wrong branch from 8867d3f to 6ec923eCompareMay 28, 2024 06:13
@seawinde

seawinde commented May 28, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@github-actions

github-actionsBot commented May 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 28, 2024
@github-actions

github-actionsBot commented May 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@seawinde
seawindeforce-pushed the fix_partition_mv_rewrite_result_wrong branch from 6ec923e to 0b55c46CompareMay 28, 2024 07:14
@seawinde

seawinde commented May 28, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented May 28, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41875 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b55c4607ae54515faca6a342048057176785467, data reload: false
------ Round 1 ----------------------------------
q1	17613	4339	4243	4243
q2	2020	195	193	193
q3	10466	1283	1227	1227
q4	10213	768	785	768
q5	7481	2702	2735	2702
q6	218	134	138	134
q7	968	622	621	621
q8	9228	2159	2081	2081
q9	9252	6773	6739	6739
q10	9637	3937	3874	3874
q11	452	251	246	246
q12	455	232	230	230
q13	17331	3218	3330	3218
q14	254	213	217	213
q15	516	459	476	459
q16	470	417	396	396
q17	984	677	755	677
q18	8478	7714	7725	7714
q19	7025	1565	1578	1565
q20	653	317	316	316
q21	5174	4101	3982	3982
q22	350	283	277	277
Total cold run time: 119238 ms
Total hot run time: 41875 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4516	4451	4416	4416
q2	383	269	277	269
q3	3215	2938	2864	2864
q4	1880	1593	1590	1590
q5	5473	5523	5501	5501
q6	217	126	129	126
q7	2204	1835	1801	1801
q8	3297	3415	3438	3415
q9	8649	8723	8692	8692
q10	3969	3838	3837	3837
q11	590	498	507	498
q12	839	643	604	604
q13	15828	3133	3197	3133
q14	297	296	285	285
q15	526	470	484	470
q16	478	444	447	444
q17	1777	1518	1492	1492
q18	7703	7617	7394	7394
q19	1663	1560	1508	1508
q20	2008	1792	1801	1792
q21	9361	4685	4671	4671
q22	569	487	483	483
Total cold run time: 75442 ms
Total hot run time: 55285 ms

@doris-robot

doris-robot commented May 28, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 168944 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0b55c4607ae54515faca6a342048057176785467, data reload: false
query1	942	387	380	380
query2	6457	2400	2331	2331
query3	6645	201	203	201
query4	19563	17536	17374	17374
query5	4168	416	410	410
query6	256	153	173	153
query7	4595	309	293	293
query8	239	176	186	176
query9	8633	2460	2466	2460
query10	443	291	276	276
query11	10605	10008	10122	10008
query12	137	94	85	85
query13	1630	369	379	369
query14	10095	6679	7552	6679
query15	226	168	164	164
query16	7703	271	263	263
query17	1689	522	537	522
query18	1812	269	270	269
query19	195	159	147	147
query20	91	86	82	82
query21	202	140	133	133
query22	4196	3851	3893	3851
query23	33863	32888	32973	32888
query24	7229	2882	2848	2848
query25	587	351	357	351
query26	704	165	158	158
query27	2167	322	335	322
query28	5242	2097	2103	2097
query29	843	609	594	594
query30	245	151	154	151
query31	982	761	751	751
query32	91	58	56	56
query33	524	283	262	262
query34	857	480	473	473
query35	718	605	585	585
query36	1068	928	941	928
query37	103	65	68	65
query38	2897	2749	2785	2749
query39	832	799	784	784
query40	195	127	121	121
query41	47	44	44	44
query42	105	100	97	97
query43	575	509	554	509
query44	1051	733	741	733
query45	179	163	161	161
query46	1068	717	712	712
query47	1844	1768	1776	1768
query48	366	304	307	304
query49	839	381	401	381
query50	771	390	403	390
query51	6989	6787	6690	6690
query52	108	95	93	93
query53	359	286	296	286
query54	563	477	436	436
query55	74	72	74	72
query56	278	249	250	249
query57	1142	1037	1010	1010
query58	231	202	202	202
query59	3430	3202	2956	2956
query60	272	257	264	257
query61	89	92	89	89
query62	600	454	470	454
query63	310	291	293	291
query64	8463	2321	1744	1744
query65	3171	3106	3116	3106
query66	772	326	330	326
query67	15183	15028	14807	14807
query68	5322	556	554	554
query69	499	271	269	269
query70	1194	1129	1094	1094
query71	443	275	267	267
query72	7745	2710	2558	2558
query73	751	329	326	326
query74	6043	5690	5590	5590
query75	3556	2595	2689	2595
query76	3260	951	965	951
query77	557	268	271	268
query78	10197	9793	9774	9774
query79	1175	529	519	519
query80	668	438	444	438
query81	506	217	219	217
query82	623	91	89	89
query83	195	169	169	169
query84	250	86	84	84
query85	1297	280	264	264
query86	468	308	308	308
query87	3259	3166	3113	3113
query88	3922	2465	2449	2449
query89	487	397	390	390
query90	2002	194	191	191
query91	128	98	99	98
query92	60	51	51	51
query93	1763	526	514	514
query94	1212	184	195	184
query95	479	320	314	314
query96	592	275	271	271
query97	3206	2991	2990	2990
query98	245	221	218	218
query99	1180	868	851	851
Total cold run time: 261832 ms
Total hot run time: 168944 ms

@doris-robot

doris-robot commented May 28, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0b55c4607ae54515faca6a342048057176785467, data reload: false
query1	0.04	0.03	0.04
query2	0.09	0.04	0.05
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.52	0.48	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.55	0.54
query11	0.16	0.12	0.11
query12	0.16	0.12	0.13
query13	0.60	0.58	0.60
query14	0.77	0.80	0.77
query15	0.84	0.82	0.82
query16	0.36	0.37	0.37
query17	0.98	0.94	1.01
query18	0.23	0.24	0.24
query19	1.75	1.74	1.72
query20	0.02	0.01	0.02
query21	15.52	0.70	0.67
query22	4.10	7.47	1.64
query23	18.28	1.44	1.24
query24	1.61	0.30	0.22
query25	0.14	0.09	0.09
query26	0.27	0.17	0.17
query27	0.08	0.09	0.08
query28	13.39	1.01	1.00
query29	13.19	3.35	3.28
query30	0.27	0.09	0.08
query31	2.84	0.40	0.39
query32	3.25	0.46	0.46
query33	2.84	2.91	2.87
query34	17.21	4.39	4.44
query35	4.47	4.49	4.50
query36	0.71	0.50	0.49
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.18	0.14	0.15
query41	0.10	0.04	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 109.66 s
Total hot run time: 30.3 s

zfr9527
zfr9527 approved these changes May 29, 2024
View reviewed changes

@zfr9527zfr9527 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morrySnow
morrySnow merged commit 0b5dd11 into apache:masterMay 29, 2024
yiguolei pushed a commit that referenced this pull request May 29, 2024
@seawinde@Doris-Extras
[fix](mtmv) Fix partition mv rewrite result wrong (#35236)
eefea4c
this is brought by #33800
if mv is partitioned materialzied view,
the data will be wrong by using the hited materialized view when the
paritions in related base partiton table are deleted, created and so on.
this fix the problem.
if **SET enable_materialized_view_union_rewrite=true;** this will use
the materializd view and make sure the data is corrent
if **SET enable_materialized_view_union_rewrite=false;** this will query
base table directly to make sure the data is right
dataroaring pushed a commit that referenced this pull request May 31, 2024
@seawinde@dataroaring
[fix](mtmv) Fix partition mv rewrite result wrong (#35236)
05329cf
this is brought by #33800
if mv is partitioned materialzied view,
the data will be wrong by using the hited materialized view when the
paritions in related base partiton table are deleted, created and so on.
this fix the problem.
if **SET enable_materialized_view_union_rewrite=true;** this will use
the materializd view and make sure the data is corrent
if **SET enable_materialized_view_union_rewrite=false;** this will query
base table directly to make sure the data is right
@yiguoleiyiguolei mentioned this pull request Jun 1, 2024
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@seawinde@HappenLee
[fix](mtmv) Fix partition mv rewrite result wrong (apache#35236)
472e885
this is brought by apache#33800
if mv is partitioned materialzied view,
the data will be wrong by using the hited materialized view when the
paritions in related base partiton table are deleted, created and so on.
this fix the problem.
if **SET enable_materialized_view_union_rewrite=true;** this will use
the materializd view and make sure the data is corrent
if **SET enable_materialized_view_union_rewrite=false;** this will query
base table directly to make sure the data is right
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zddrzddrzddr approved these changes
@morrySnowmorrySnowmorrySnow approved these changes
+1 more reviewer
@zfr9527zfr9527zfr9527 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.4-mergeddev/3.0.0-mergedp0_wreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@seawinde@doris-robot@morrySnow@zddr@zfr9527@dataroaring