[fix](mtmv)Fix that the storage medium specified for the mtmv is SSD, but the partition storage medium for the mtmv is still HDD by zddr · Pull Request #35644 · apache/doris

zddr · 2024-05-30T07:04:04Z

fix when create mtmv use property 'storage_medium' = 'SSD',after refresh mtmv, partition's storage_medium is HDD

Not only is there a problem with mtmv, but essentially, when creating partitions, there is no inheritance of the table

doris-robot · 2024-05-30T07:04:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zddr · 2024-05-30T07:09:01Z

run buildall

zddr · 2024-05-30T07:19:41Z

run performance

zddr · 2024-05-30T09:51:57Z

run buildall

doris-robot · 2024-05-30T10:30:15Z

TPC-H: Total hot run time: 41674 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26339d2c090e1465beb0529e503fa2d0c4a49e78, data reload: false
------ Round 1 ----------------------------------
q1	17612	4785	4234	4234
q2	2025	209	200	200
q3	10452	1219	1184	1184
q4	10195	806	742	742
q5	7484	2726	2558	2558
q6	232	131	132	131
q7	984	637	596	596
q8	9219	2127	2076	2076
q9	8962	6696	6643	6643
q10	9366	3948	3883	3883
q11	459	241	239	239
q12	513	241	225	225
q13	17924	3144	3215	3144
q14	251	228	217	217
q15	518	471	478	471
q16	488	391	398	391
q17	983	694	646	646
q18	8186	7850	7971	7850
q19	2868	1665	1599	1599
q20	673	321	313	313
q21	5244	3986	4058	3986
q22	396	346	361	346
Total cold run time: 115034 ms
Total hot run time: 41674 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4555	4351	4394	4351
q2	362	277	267	267
q3	3129	2972	2776	2776
q4	1839	1642	1647	1642
q5	5436	5449	5468	5449
q6	218	125	121	121
q7	2170	1789	1819	1789
q8	3227	3360	3405	3360
q9	8681	8651	8581	8581
q10	3899	3842	3817	3817
q11	600	489	524	489
q12	785	620	620	620
q13	17382	3158	3208	3158
q14	290	259	270	259
q15	510	468	467	467
q16	463	403	419	403
q17	1769	1478	1470	1470
q18	7760	7633	7533	7533
q19	1683	1538	1585	1538
q20	1996	1781	1769	1769
q21	10105	4704	4667	4667
q22	610	519	519	519
Total cold run time: 77469 ms
Total hot run time: 55045 ms

doris-robot · 2024-05-30T10:41:05Z

TPC-DS: Total hot run time: 172287 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26339d2c090e1465beb0529e503fa2d0c4a49e78, data reload: false
query1	930	388	372	372
query2	6454	2327	2536	2327
query3	6651	210	215	210
query4	19914	17357	17307	17307
query5	4115	423	430	423
query6	253	154	151	151
query7	4579	302	289	289
query8	321	298	286	286
query9	8475	2394	2368	2368
query10	446	275	252	252
query11	10657	10034	10170	10034
query12	137	89	90	89
query13	1644	386	373	373
query14	10103	7007	7580	7007
query15	260	193	190	190
query16	7724	271	271	271
query17	1686	560	542	542
query18	1784	283	280	280
query19	205	163	158	158
query20	99	93	92	92
query21	203	136	131	131
query22	4231	3980	3853	3853
query23	33564	33131	32840	32840
query24	10533	2882	2834	2834
query25	573	372	359	359
query26	702	157	153	153
query27	2204	323	323	323
query28	5784	2089	2077	2077
query29	874	602	610	602
query30	251	150	153	150
query31	982	758	731	731
query32	90	54	53	53
query33	653	293	260	260
query34	870	487	487	487
query35	726	612	598	598
query36	1057	934	898	898
query37	105	64	65	64
query38	2922	2805	2785	2785
query39	858	803	800	800
query40	191	121	119	119
query41	53	49	50	49
query42	105	96	98	96
query43	588	540	543	540
query44	1099	744	756	744
query45	186	166	167	166
query46	1089	753	724	724
query47	1846	1776	1813	1776
query48	359	298	291	291
query49	839	380	373	373
query50	772	382	386	382
query51	6846	6723	6640	6640
query52	103	92	90	90
query53	354	288	291	288
query54	892	429	436	429
query55	78	73	75	73
query56	262	234	244	234
query57	1113	1068	1052	1052
query58	219	220	208	208
query59	3339	3249	3026	3026
query60	291	257	247	247
query61	86	84	84	84
query62	617	442	463	442
query63	311	290	301	290
query64	8525	2210	1757	1757
query65	3160	3077	3139	3077
query66	797	325	335	325
query67	15439	14879	14798	14798
query68	4517	553	543	543
query69	444	270	272	270
query70	1140	1076	1157	1076
query71	395	266	272	266
query72	7569	5700	5525	5525
query73	716	327	324	324
query74	6013	5698	5635	5635
query75	3333	2592	2626	2592
query76	2459	1067	1077	1067
query77	381	268	263	263
query78	10319	10094	9802	9802
query79	2318	516	513	513
query80	1098	439	502	439
query81	564	226	222	222
query82	673	93	90	90
query83	254	175	173	173
query84	249	84	89	84
query85	1835	270	266	266
query86	509	323	309	309
query87	3319	3135	3113	3113
query88	3953	2349	2345	2345
query89	495	402	374	374
query90	2029	189	191	189
query91	127	95	99	95
query92	62	49	50	49
query93	2089	518	504	504
query94	1196	237	183	183
query95	409	314	320	314
query96	587	260	263	260
query97	3167	3007	3000	3000
query98	248	225	213	213
query99	1184	850	863	850
Total cold run time: 266829 ms
Total hot run time: 172287 ms

doris-robot · 2024-05-30T10:46:34Z

ClickBench: Total hot run time: 30.88 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26339d2c090e1465beb0529e503fa2d0c4a49e78, data reload: false
query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.23	0.04	0.04
query4	1.69	0.07	0.07
query5	0.51	0.49	0.50
query6	1.12	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.54	0.48	0.49
query10	0.54	0.55	0.53
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.79	0.77	0.78
query15	0.83	0.81	0.81
query16	0.38	0.36	0.36
query17	0.97	0.96	0.96
query18	0.23	0.26	0.24
query19	1.78	1.78	1.66
query20	0.01	0.01	0.01
query21	15.56	0.67	0.65
query22	3.94	6.50	2.37
query23	18.32	1.31	1.27
query24	1.45	0.26	0.26
query25	0.14	0.08	0.09
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.52	1.02	0.99
query29	13.20	3.34	3.22
query30	0.24	0.05	0.06
query31	2.88	0.37	0.37
query32	3.31	0.46	0.47
query33	2.90	2.92	2.90
query34	16.98	4.40	4.47
query35	4.51	4.50	4.48
query36	0.71	0.50	0.47
query37	0.18	0.16	0.15
query38	0.15	0.15	0.16
query39	0.04	0.04	0.03
query40	0.16	0.13	0.15
query41	0.09	0.05	0.04
query42	0.06	0.04	0.04
query43	0.03	0.03	0.04
Total cold run time: 109.43 s
Total hot run time: 30.88 s

zddr · 2024-05-31T07:46:20Z

run buildall

doris-robot · 2024-05-31T08:11:58Z

TPC-H: Total hot run time: 40159 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 034c3c3ad3b6ba2e500824511b8ca6a54eb3fca5, data reload: false
------ Round 1 ----------------------------------
q1	6590	4284	4312	4284
q2	1685	197	199	197
q3	1441	1042	1124	1042
q4	4715	878	726	726
q5	7367	2794	2843	2794
q6	241	150	147	147
q7	1024	621	598	598
q8	2040	2065	2072	2065
q9	9241	6543	6556	6543
q10	8760	3733	3694	3694
q11	373	239	232	232
q12	405	220	212	212
q13	4808	3062	3009	3009
q14	253	220	227	220
q15	520	489	474	474
q16	475	433	413	413
q17	963	660	709	660
q18	8172	7586	7455	7455
q19	1663	1566	1441	1441
q20	505	312	315	312
q21	5220	3309	3951	3309
q22	402	342	332	332
Total cold run time: 66863 ms
Total hot run time: 40159 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4374	4310	4227	4227
q2	372	259	279	259
q3	2968	2775	2771	2771
q4	1916	1658	1669	1658
q5	5299	5291	5296	5291
q6	218	125	123	123
q7	2127	1753	1686	1686
q8	3275	3313	3311	3311
q9	8363	8357	8368	8357
q10	3978	3681	3661	3661
q11	569	481	481	481
q12	764	578	591	578
q13	3485	3021	2997	2997
q14	297	281	258	258
q15	512	477	480	477
q16	476	419	438	419
q17	1760	1489	1480	1480
q18	7623	7509	7540	7509
q19	1673	1616	1585	1585
q20	1979	1779	1725	1725
q21	4927	4657	4656	4656
q22	636	538	527	527
Total cold run time: 57591 ms
Total hot run time: 54036 ms

zddr · 2024-06-03T01:58:00Z

run buildall

doris-robot · 2024-06-03T02:20:44Z

TPC-H: Total hot run time: 41718 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dfe8fd1aec07ed7ed09ed572c5d2229d68572809, data reload: false
------ Round 1 ----------------------------------
q1	17708	4401	4285	4285
q2	2040	193	186	186
q3	10531	1326	1085	1085
q4	10221	743	769	743
q5	7492	2696	2668	2668
q6	229	140	142	140
q7	944	630	624	624
q8	9214	2102	2096	2096
q9	9494	6682	6686	6682
q10	9649	3869	3945	3869
q11	449	235	234	234
q12	492	230	231	230
q13	17362	3245	3210	3210
q14	282	227	229	227
q15	504	485	487	485
q16	506	399	396	396
q17	1007	688	695	688
q18	8262	7853	7849	7849
q19	7965	1357	1449	1357
q20	711	313	310	310
q21	5112	4015	4026	4015
q22	397	339	345	339
Total cold run time: 120571 ms
Total hot run time: 41718 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4533	4369	4488	4369
q2	383	279	268	268
q3	3101	2973	2866	2866
q4	2000	1760	1611	1611
q5	5301	5474	5473	5473
q6	219	125	123	123
q7	2214	1807	1814	1807
q8	3254	3372	3341	3341
q9	8667	8601	8608	8601
q10	4086	3870	3832	3832
q11	591	503	485	485
q12	768	587	589	587
q13	16164	3063	3138	3063
q14	318	275	272	272
q15	520	482	504	482
q16	484	416	443	416
q17	1781	1487	1522	1487
q18	8084	7574	7338	7338
q19	1742	1622	1528	1528
q20	2778	1774	1800	1774
q21	6486	4760	4816	4760
q22	745	564	529	529
Total cold run time: 74219 ms
Total hot run time: 55012 ms

doris-robot · 2024-06-03T02:31:36Z

TPC-DS: Total hot run time: 169392 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dfe8fd1aec07ed7ed09ed572c5d2229d68572809, data reload: false
query1	928	388	377	377
query2	6441	2324	2399	2324
query3	6633	203	206	203
query4	19404	17493	17163	17163
query5	4126	454	436	436
query6	238	158	150	150
query7	4576	300	305	300
query8	319	283	299	283
query9	8270	2352	2341	2341
query10	436	294	272	272
query11	10552	10096	9979	9979
query12	138	92	89	89
query13	1631	366	354	354
query14	10191	7011	7406	7011
query15	239	188	187	187
query16	7631	263	269	263
query17	1298	519	501	501
query18	1933	268	265	265
query19	193	150	150	150
query20	97	87	85	85
query21	208	130	126	126
query22	4261	4155	3990	3990
query23	33650	33043	32823	32823
query24	12021	2814	2798	2798
query25	658	358	378	358
query26	1789	150	152	150
query27	3022	312	325	312
query28	7235	2009	2020	2009
query29	1154	616	604	604
query30	228	152	153	152
query31	959	747	766	747
query32	94	55	54	54
query33	759	286	281	281
query34	945	479	466	466
query35	727	617	617	617
query36	1089	934	965	934
query37	200	69	72	69
query38	2896	2749	2745	2745
query39	868	797	823	797
query40	285	124	125	124
query41	55	51	51	51
query42	119	92	100	92
query43	558	568	545	545
query44	1246	741	745	741
query45	203	168	173	168
query46	1062	728	737	728
query47	1836	1754	1780	1754
query48	389	302	298	298
query49	1044	409	407	407
query50	788	379	391	379
query51	6886	6800	6673	6673
query52	103	89	95	89
query53	358	289	289	289
query54	919	497	446	446
query55	73	73	73	73
query56	298	248	259	248
query57	1143	1046	1067	1046
query58	258	250	243	243
query59	3321	3258	3148	3148
query60	305	283	263	263
query61	90	90	83	83
query62	639	433	438	433
query63	311	289	292	289
query64	9912	2182	1706	1706
query65	3214	3109	3100	3100
query66	1354	322	329	322
query67	15237	14909	15377	14909
query68	4474	554	545	545
query69	572	429	406	406
query70	1157	1048	1068	1048
query71	421	281	277	277
query72	6921	5582	2686	2686
query73	760	329	322	322
query74	5898	5514	5492	5492
query75	3371	2681	2701	2681
query76	2374	941	944	941
query77	440	287	282	282
query78	10282	9877	9716	9716
query79	1502	514	517	514
query80	2075	449	451	449
query81	588	218	216	216
query82	1014	107	101	101
query83	296	162	165	162
query84	266	83	81	81
query85	1381	272	256	256
query86	494	318	292	292
query87	3234	3082	3057	3057
query88	3380	2420	2420	2420
query89	472	396	399	396
query90	1840	187	186	186
query91	126	100	96	96
query92	66	49	53	49
query93	1691	510	501	501
query94	1395	185	181	181
query95	406	305	306	305
query96	597	268	266	266
query97	3144	3028	3061	3028
query98	257	223	215	215
query99	1270	813	848	813
Total cold run time: 272207 ms
Total hot run time: 169392 ms

doris-robot · 2024-06-03T02:37:06Z

ClickBench: Total hot run time: 30.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dfe8fd1aec07ed7ed09ed572c5d2229d68572809, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.05	0.05
query4	1.68	0.06	0.06
query5	0.53	0.48	0.49
query6	1.13	0.74	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.54	0.47	0.48
query10	0.54	0.55	0.55
query11	0.14	0.11	0.12
query12	0.15	0.12	0.11
query13	0.60	0.59	0.60
query14	0.76	0.78	0.80
query15	0.83	0.81	0.82
query16	0.36	0.36	0.36
query17	1.02	1.03	0.96
query18	0.23	0.24	0.24
query19	1.78	1.75	1.70
query20	0.01	0.02	0.01
query21	15.51	0.68	0.68
query22	4.37	7.84	1.65
query23	18.30	1.32	1.32
query24	1.48	0.33	0.21
query25	0.15	0.08	0.09
query26	0.25	0.17	0.17
query27	0.08	0.07	0.08
query28	13.48	1.01	0.98
query29	13.34	3.28	3.27
query30	0.24	0.06	0.05
query31	2.86	0.40	0.39
query32	3.27	0.48	0.47
query33	2.87	2.96	2.91
query34	17.24	4.37	4.39
query35	4.51	4.46	4.67
query36	0.65	0.47	0.50
query37	0.18	0.15	0.16
query38	0.16	0.16	0.15
query39	0.05	0.03	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.03	0.04	0.04
Total cold run time: 110.03 s
Total hot run time: 30.25 s

github-actions · 2024-06-03T03:44:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-03T03:44:50Z

PR approved by anyone and no changes requested.

shuke987

LGTM

…, but the partition storage medium for the mtmv is still HDD (#35644) fix when create mtmv use property `'storage_medium' = 'SSD'`,after refresh mtmv, partition's `storage_medium` is HDD Not only is there a problem with mtmv, but essentially, when creating partitions, there is no inheritance of the table

…, but the partition storage medium for the mtmv is still HDD (apache#35644) fix when create mtmv use property `'storage_medium' = 'SSD'`,after refresh mtmv, partition's `storage_medium` is HDD Not only is there a problem with mtmv, but essentially, when creating partitions, there is no inheritance of the table

…, but the partition storage medium for the mtmv is still HDD (#35644) (#35955) pick from master:#35644

… table (#41192) revert #35644 table if not set storage medium, will set default by `Config.default_storage_medium`, at present, we cannot distinguish whether the storage medium of the table is set by the user. If it is not set by the user, it represents priority use rather than mandatory use. If we directly inherit the storage medium of the table, it may cause the partition to be forced to use 'Config. default_storage_madium', resulting in the partition being unable to be created

… table (apache#41192) revert apache#35644 table if not set storage medium, will set default by `Config.default_storage_medium`, at present, we cannot distinguish whether the storage medium of the table is set by the user. If it is not set by the user, it represents priority use rather than mandatory use. If we directly inherit the storage medium of the table, it may cause the partition to be forced to use 'Config. default_storage_madium', resulting in the partition being unable to be created

… table (#41192) revert #35644 table if not set storage medium, will set default by `Config.default_storage_medium`, at present, we cannot distinguish whether the storage medium of the table is set by the user. If it is not set by the user, it represents priority use rather than mandatory use. If we directly inherit the storage medium of the table, it may cause the partition to be forced to use 'Config. default_storage_madium', resulting in the partition being unable to be created

…, but the partition storage medium for the mtmv is still HDD (apache#35644) fix when create mtmv use property `'storage_medium' = 'SSD'`,after refresh mtmv, partition's `storage_medium` is HDD Not only is there a problem with mtmv, but essentially, when creating partitions, there is no inheritance of the table

… table (apache#41192) revert apache#35644 table if not set storage medium, will set default by `Config.default_storage_medium`, at present, we cannot distinguish whether the storage medium of the table is set by the user. If it is not set by the user, it represents priority use rather than mandatory use. If we directly inherit the storage medium of the table, it may cause the partition to be forced to use 'Config. default_storage_madium', resulting in the partition being unable to be created

问题描述：创建异步物化视图（MTMV）时指定 storage_medium，如： CREATE MATERIALIZED VIEW mv ... PROPERTIES ('storage_medium' = 'SSD') REFRESH 后，基表新增分区在 MTMV 中对齐创建的新分区，其 storage_medium 始终为 HDD（Config.default_storage_medium），而非用户显式指定的 SSD。此前曾尝试在 InternalCatalog.addPartition() 公共层解决此问题（apache#35644），但该方案无法区分"用户未设置 storage_medium"与 "用户显式设置为默认值"两种场景，导致普通表分区也被强制绑定默认介质、创建失败，后被回退（apache#41192）。核心困难在于 InternalCatalog 层拿到的 OlapTable.getStorageMedium() 不会区分"显式设置"与"未设置"。本方案仅在 MTMV 刷新路径生效（MTMVPartitionUtil.addPartition），不涉及普通表 ADD PARTITION 行为。利用 OlapTable.getStorageMedium() 在 TableProperty 未设置 storage_medium 时返回 null 的特性，配合 null 判空 + DEFAULT_STORAGE_MEDIUM 对比，三态逻辑准确： - getStorageMedium() == null → user 未设，跳过 - 等于 DEFAULT_STORAGE_MEDIUM → 显式设置为默认值，跳过 - 不等于 DEFAULT_STORAGE_MEDIUM → 显式设置为非默认值，覆盖

zddr added 3 commits May 29, 2024 18:25

1
6a9aa9a

Merge branch 'master' into mv_storage_medium
bc4f459

1
3d04f25

morrySnow added the dev/2.1.x label May 30, 2024

1
26339d2

Merge branch 'master' into mv_storage_medium
034c3c3

Merge branch 'master' into mv_storage_medium
dfe8fd1

morrySnow approved these changes Jun 3, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 3, 2024

github-actionsBot added the reviewed label Jun 3, 2024

seawinde approved these changes Jun 3, 2024
View reviewed changes

shuke987 approved these changes Jun 3, 2024
View reviewed changes

morrySnow merged commit 1113f3d into apache:masterJun 3, 2024

zddr mentioned this pull request Jun 6, 2024
[fix](mtmv) Fix that the storage medium specified for the mtmv is SSD, but the partition storage medium for the mtmv is still HDD (#35644) #35955
Merged

morningman pushed a commit that referenced this pull request Jun 6, 2024
[fix](mtmv) Fix that the storage medium specified for the mtmv is SSD…
cd808c3
…, but the partition storage medium for the mtmv is still HDD (#35644) (#35955) pick from master:#35644

morningman added dev/2.1.4-merged and removed dev/2.1.x labels Jun 6, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

dataroaring added the kind/behavior-changed label Sep 23, 2024

morrySnow mentioned this pull request Sep 24, 2024
[revert](storage) storage medium of partition should not inherit from table #41192
Merged

Uh oh!

Conversation

zddr commented May 30, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

zddr commented May 30, 2024

Uh oh!

zddr commented May 30, 2024

Uh oh!

zddr commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

zddr commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

zddr commented Jun 3, 2024

Uh oh!

doris-robot commented Jun 3, 2024

Uh oh!

doris-robot commented Jun 3, 2024

Uh oh!

doris-robot commented Jun 3, 2024

Uh oh!

github-actionsBot commented Jun 3, 2024

Uh oh!

github-actionsBot commented Jun 3, 2024

Uh oh!

shuke987 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zddr commented May 30, 2024•
edited
Loading