Skip to content

[BUG] fix partition storage policy info lost - #38700

Merged
yiguolei merged 3 commits into
apache:masterfrom
qzsee:doris/fix-partition-policy
Aug 21, 2024
Merged

[BUG] fix partition storage policy info lost#38700
yiguolei merged 3 commits into
apache:masterfrom
qzsee:doris/fix-partition-policy

Conversation

@qzsee

@qzseeqzsee commented Aug 1, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

  1. fix partition storage policy info lost
    When adding a storage policy to a table through an alter statement, the partition policy is lost when the FE is restarted because the storage policy is not set for the partition synchronously.

  2. when setting policies, check the uniq table in advance to prevent metadata inconsistencies

  3. show storage policy using for stmt support any string policy name

    If the policy name begins with a number, the statement cannot be parsed.

Issue Number: close #xxx

@doris-robot

doris-robot commented Aug 1, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qzsee

qzsee commented Aug 1, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 1, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 42044 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a48de9aef4113ec881160efe98a22d4adf33979d, data reload: false
------ Round 1 ----------------------------------
q1	17623	4227	4111	4111
q2	2018	207	218	207
q3	11217	1359	1397	1359
q4	10246	844	1035	844
q5	8589	3108	3036	3036
q6	221	137	139	137
q7	1060	631	628	628
q8	11169	1906	2004	1906
q9	8594	6617	6616	6616
q10	8787	3871	3874	3871
q11	435	249	248	248
q12	415	228	231	228
q13	17768	2954	2960	2954
q14	274	247	244	244
q15	523	487	498	487
q16	529	385	391	385
q17	980	887	895	887
q18	8391	7350	7293	7293
q19	1389	1224	1223	1223
q20	568	309	348	309
q21	5351	4787	4867	4787
q22	359	288	284	284
Total cold run time: 116506 ms
Total hot run time: 42044 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4072	4024	4044	4024
q2	332	234	222	222
q3	3133	3158	3215	3158
q4	2019	2057	2005	2005
q5	5487	5440	5411	5411
q6	218	130	132	130
q7	2217	1842	1804	1804
q8	3309	3366	3378	3366
q9	8823	8734	8572	8572
q10	3997	4042	4035	4035
q11	579	465	474	465
q12	803	586	569	569
q13	11402	3124	3156	3124
q14	321	300	292	292
q15	550	486	486	486
q16	468	416	426	416
q17	1769	1752	1747	1747
q18	8461	7782	7826	7782
q19	1729	1716	1716	1716
q20	2099	1754	1751	1751
q21	5672	5472	5485	5472
q22	523	481	473	473
Total cold run time: 67983 ms
Total hot run time: 57020 ms

@doris-robot

doris-robot commented Aug 1, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 170492 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a48de9aef4113ec881160efe98a22d4adf33979d, data reload: false
query1	923	376	371	371
query2	6295	1779	1780	1779
query3	6636	218	229	218
query4	19923	17470	17335	17335
query5	3789	518	538	518
query6	268	184	173	173
query7	4609	303	291	291
query8	260	206	199	199
query9	8525	2387	2372	2372
query10	449	294	268	268
query11	10446	10037	10083	10037
query12	128	93	90	90
query13	1653	393	375	375
query14	9576	7046	7268	7046
query15	209	167	163	163
query16	7086	476	462	462
query17	953	599	585	585
query18	1925	303	291	291
query19	198	157	150	150
query20	98	88	92	88
query21	208	103	104	103
query22	4302	4023	4166	4023
query23	33944	33858	33214	33214
query24	10228	3146	3153	3146
query25	719	439	443	439
query26	1781	161	162	161
query27	2904	291	300	291
query28	7378	2054	2023	2023
query29	1333	492	456	456
query30	235	162	157	157
query31	975	779	800	779
query32	105	59	60	59
query33	693	346	354	346
query34	950	528	530	528
query35	899	784	761	761
query36	1109	937	903	903
query37	293	93	90	90
query38	3004	2796	2825	2796
query39	873	908	803	803
query40	251	120	123	120
query41	45	43	44	43
query42	124	103	103	103
query43	504	456	436	436
query44	1199	741	741	741
query45	209	177	184	177
query46	1091	830	838	830
query47	1867	1705	1682	1682
query48	379	295	295	295
query49	936	433	443	433
query50	899	442	446	442
query51	6939	6900	6746	6746
query52	109	94	90	90
query53	269	189	188	188
query54	639	465	452	452
query55	81	76	77	76
query56	278	262	254	254
query57	1130	1036	1062	1036
query58	293	283	273	273
query59	2676	2384	2585	2384
query60	290	289	273	273
query61	97	94	97	94
query62	896	680	652	652
query63	220	191	187	187
query64	5557	1932	1879	1879
query65	3168	3129	3086	3086
query66	1293	338	332	332
query67	15301	14907	15122	14907
query68	4711	574	628	574
query69	705	387	336	336
query70	1155	1064	1086	1064
query71	529	287	287	287
query72	8032	2743	2504	2504
query73	794	338	332	332
query74	6095	5697	5662	5662
query75	4356	2734	2726	2726
query76	3982	1260	1317	1260
query77	686	317	328	317
query78	9468	8936	8896	8896
query79	2989	531	552	531
query80	2310	539	538	538
query81	569	235	224	224
query82	1370	140	137	137
query83	291	178	175	175
query84	270	90	82	82
query85	1459	315	349	315
query86	463	304	287	287
query87	3319	3102	3100	3100
query88	3712	2401	2420	2401
query89	402	290	308	290
query90	2102	198	202	198
query91	129	103	105	103
query92	67	54	57	54
query93	2952	621	628	621
query94	982	272	300	272
query95	387	270	273	270
query96	616	284	284	284
query97	3241	3087	3060	3060
query98	215	211	204	204
query99	1583	1288	1313	1288
Total cold run time: 272563 ms
Total hot run time: 170492 ms

@doris-robot

doris-robot commented Aug 1, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a48de9aef4113ec881160efe98a22d4adf33979d, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.05
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.48	0.47	0.48
query6	1.16	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.56	0.51	0.51
query10	0.57	0.57	0.56
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.61	0.60	0.60
query14	0.76	0.80	0.78
query15	0.89	0.86	0.88
query16	0.36	0.36	0.35
query17	1.02	0.98	1.04
query18	0.23	0.21	0.21
query19	1.82	1.72	1.75
query20	0.01	0.00	0.01
query21	15.78	0.76	0.67
query22	4.09	8.18	1.25
query23	18.06	1.26	1.35
query24	2.26	0.22	0.22
query25	0.18	0.08	0.08
query26	0.32	0.21	0.21
query27	0.46	0.24	0.23
query28	13.16	1.00	0.99
query29	12.56	3.31	3.30
query30	0.26	0.06	0.05
query31	2.86	0.41	0.40
query32	3.24	0.50	0.48
query33	2.92	2.92	2.94
query34	15.49	4.26	4.30
query35	4.33	4.32	4.35
query36	0.68	0.48	0.49
query37	0.18	0.17	0.15
query38	0.18	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.14	0.12
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.25 s
Total hot run time: 29.94 s

@morningmanmorningman self-assigned this Aug 2, 2024
@morningmanmorningman added usercase Important user case type label dev/3.0.x labels Aug 12, 2024
@qzsee

qzsee commented Aug 12, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

morningman
morningman approved these changes Aug 12, 2024
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Aug 12, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 12, 2024
@github-actions

github-actionsBot commented Aug 12, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman

morningman commented Aug 12, 2024

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Aug 12, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39749 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 34de4b969b129d92fdb4e06a36fe4c336fe959ba, data reload: false
------ Round 1 ----------------------------------
q1	18220	4421	4378	4378
q2	2919	179	183	179
q3	10714	1148	1162	1148
q4	10478	796	807	796
q5	7538	2561	2508	2508
q6	226	143	142	142
q7	970	610	615	610
q8	9325	1932	1940	1932
q9	8796	6549	6549	6549
q10	7057	2210	2217	2210
q11	463	246	246	246
q12	395	220	225	220
q13	18757	2979	3003	2979
q14	274	245	247	245
q15	531	491	488	488
q16	511	385	402	385
q17	980	670	676	670
q18	8026	7538	7356	7356
q19	4074	1092	1104	1092
q20	700	318	341	318
q21	5350	4398	4271	4271
q22	1126	1027	1033	1027
Total cold run time: 117430 ms
Total hot run time: 39749 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4392	4308	4243	4243
q2	379	279	272	272
q3	2822	2567	2585	2567
q4	1880	1607	1666	1607
q5	5263	5322	5287	5287
q6	213	132	133	132
q7	2053	1686	1638	1638
q8	3151	3329	3277	3277
q9	8428	8358	8388	8358
q10	3372	3143	3177	3143
q11	579	498	489	489
q12	793	614	602	602
q13	17559	2987	2982	2982
q14	310	271	272	271
q15	535	485	480	480
q16	465	414	443	414
q17	1775	1483	1452	1452
q18	7613	7470	7383	7383
q19	1657	1546	1550	1546
q20	1968	1771	1775	1771
q21	5257	4990	5093	4990
q22	1088	1041	1013	1013
Total cold run time: 71552 ms
Total hot run time: 53917 ms

@doris-robot

doris-robot commented Aug 12, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 201023 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 34de4b969b129d92fdb4e06a36fe4c336fe959ba, data reload: false
query1	912	364	355	355
query2	6445	1909	1886	1886
query3	6650	210	219	210
query4	31083	23188	23041	23041
query5	4215	505	479	479
query6	274	192	168	168
query7	4582	298	297	297
query8	254	198	192	192
query9	8522	2518	2476	2476
query10	552	436	456	436
query11	15650	14873	15024	14873
query12	153	97	97	97
query13	1647	368	367	367
query14	10216	7553	7635	7553
query15	279	215	228	215
query16	7827	470	474	470
query17	1736	550	524	524
query18	1970	271	281	271
query19	188	145	141	141
query20	113	103	105	103
query21	205	126	102	102
query22	4195	4219	3937	3937
query23	33832	33384	33162	33162
query24	11886	2648	2642	2642
query25	653	363	358	358
query26	1776	152	155	152
query27	2960	286	281	281
query28	7396	2042	2042	2042
query29	1136	400	404	400
query30	311	148	140	140
query31	977	768	749	749
query32	93	52	55	52
query33	759	290	275	275
query34	963	458	478	458
query35	983	821	818	818
query36	1094	955	895	895
query37	290	81	84	81
query38	4324	4053	4122	4053
query39	1436	1436	1371	1371
query40	272	120	113	113
query41	48	46	44	44
query42	118	98	99	98
query43	515	454	460	454
query44	1196	728	726	726
query45	227	208	200	200
query46	1085	722	755	722
query47	1844	1770	1747	1747
query48	367	303	298	298
query49	1207	440	416	416
query50	810	409	444	409
query51	6753	6698	6560	6560
query52	105	96	92	92
query53	260	190	183	183
query54	1014	444	444	444
query55	78	77	75	75
query56	261	246	247	246
query57	1166	1053	1087	1053
query58	244	221	231	221
query59	3000	2856	2714	2714
query60	302	266	268	266
query61	96	93	93	93
query62	869	625	654	625
query63	217	187	182	182
query64	10524	2259	1757	1757
query65	3196	3139	3154	3139
query66	1378	332	328	328
query67	15320	14803	14600	14600
query68	4558	543	554	543
query69	429	377	418	377
query70	1199	1106	1156	1106
query71	404	280	277	277
query72	18634	17108	17021	17021
query73	771	327	329	327
query74	9139	8736	8725	8725
query75	3526	2684	2666	2666
query76	2807	943	978	943
query77	467	317	306	306
query78	10571	10090	8957	8957
query79	2042	526	535	526
query80	1743	497	491	491
query81	587	219	222	219
query82	601	135	132	132
query83	262	154	145	145
query84	263	76	78	76
query85	1746	292	326	292
query86	488	297	302	297
query87	4670	4519	4638	4519
query88	4040	2478	2498	2478
query89	388	291	289	289
query90	1846	194	195	194
query91	121	98	93	93
query92	70	51	51	51
query93	1524	542	533	533
query94	959	302	311	302
query95	365	280	268	268
query96	606	284	282	282
query97	3363	3160	3228	3160
query98	223	208	208	208
query99	1620	1293	1304	1293
Total cold run time: 311041 ms
Total hot run time: 201023 ms

@doris-robot

doris-robot commented Aug 12, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 34de4b969b129d92fdb4e06a36fe4c336fe959ba, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.08	0.07
query5	0.50	0.49	0.49
query6	1.14	0.73	0.73
query7	0.01	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.50	0.48
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.59	0.60	0.57
query14	0.76	0.78	0.76
query15	0.85	0.81	0.82
query16	0.37	0.36	0.36
query17	0.99	0.97	1.07
query18	0.23	0.22	0.21
query19	1.86	1.82	1.72
query20	0.01	0.01	0.01
query21	15.47	0.72	0.64
query22	4.15	7.14	2.12
query23	18.32	1.29	1.26
query24	2.08	0.24	0.22
query25	0.15	0.08	0.07
query26	0.30	0.21	0.21
query27	0.47	0.23	0.22
query28	13.25	1.03	0.99
query29	12.60	3.33	3.26
query30	0.24	0.05	0.04
query31	2.88	0.39	0.39
query32	3.26	0.49	0.46
query33	2.89	2.96	2.87
query34	17.03	4.29	4.38
query35	4.45	4.42	4.52
query36	0.66	0.47	0.47
query37	0.19	0.15	0.16
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.73 s
Total hot run time: 30.65 s

zhannngchen
zhannngchen approved these changes Aug 13, 2024
View reviewed changes

@zhannngchenzhannngchen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
yiguolei merged commit df173ae into apache:masterAug 21, 2024
qzsee added a commit to qzsee/incubator-doris that referenced this pull request Aug 21, 2024
@qzseegarenshi
[BUG] fix partition storage policy info lost (apache#38700)
51af468
## Proposed changes
1. fix partition storage policy info lost
When adding a storage policy to a table through an alter statement, the
partition policy is lost when the FE is restarted because the storage
policy is not set for the partition synchronously.
2. when setting policies, check the uniq table in advance to prevent
metadata inconsistencies
3. show storage policy using for stmt support any string policy name
If the policy name begins with a number, the statement cannot be parsed.
Issue Number: close #xxx
<!--Describe your changes.-->
---------
Co-authored-by: garenshi <garenshi@tencent.com>
@qzseeqzsee mentioned this pull request Aug 21, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 21, 2024
@qzseegarenshi
[BUG] fix partition storage policy info lost (#38700) (#39677)
76b7a52
## Proposed changes
cherry-pick from #38700 Issue Number: close #xxx
<!--Describe your changes.-->
---------
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Co-authored-by: garenshi <garenshi@tencent.com>
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
@qzseegarenshi@dataroaring
[BUG] fix partition storage policy info lost (#38700)
3d210d0
## Proposed changes
1. fix partition storage policy info lost
When adding a storage policy to a table through an alter statement, the
partition policy is lost when the FE is restarted because the storage
policy is not set for the partition synchronously.
2. when setting policies, check the uniq table in advance to prevent
metadata inconsistencies
3. show storage policy using for stmt support any string policy name
If the policy name begins with a number, the statement cannot be parsed.
Issue Number: close #xxx
<!--Describe your changes.-->
---------
Co-authored-by: garenshi <garenshi@tencent.com>
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@qzseegarenshi@HappenLee
[BUG] fix partition storage policy info lost (apache#38700)
b336ac7
## Proposed changes
1. fix partition storage policy info lost
When adding a storage policy to a table through an alter statement, the
partition policy is lost when the FE is restarted because the storage
policy is not set for the partition synchronously.
2. when setting policies, check the uniq table in advance to prevent
metadata inconsistencies
3. show storage policy using for stmt support any string policy name
If the policy name begins with a number, the statement cannot be parsed.
Issue Number: close #xxx
<!--Describe your changes.-->
---------
Co-authored-by: garenshi <garenshi@tencent.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@zhannngchenzhannngchenzhannngchen approved these changes
@dataroaringdataroaringAwaiting requested review from dataroaringdataroaring is a code owner
@CalvinKirsCalvinKirsAwaiting requested review from CalvinKirsCalvinKirs is a code owner

Assignees

@morningmanmorningman

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.6-mergeddev/3.0.2-mergeddoingreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@qzsee@doris-robot@morningman@zhannngchen@yiguolei@dataroaring