[enhancement](auto-partition) Forbid use Auto and Dynamic partition at the same time by zclllyybb · Pull Request #33736 · apache/doris

zclllyybb · 2024-04-16T22:32:14Z

Proposed changes

Issue Number: close #xxx

Forbid use Auto and Dynamic partition at the same time
Fix lost check in legacy planner for creating auto partition table

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-16T22:32:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zclllyybb · 2024-04-16T22:32:20Z

run buildall

zclllyybb · 2024-04-16T23:02:23Z

run buildall

zclllyybb · 2024-04-19T08:54:36Z

run buildall

doris-robot · 2024-04-19T09:34:52Z

TPC-H: Total hot run time: 38478 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0079bd0866030c6656cf4b21bcb757949210fb13, data reload: false
------ Round 1 ----------------------------------
q1	17610	4236	4218	4218
q2	2003	185	182	182
q3	10469	1100	1098	1098
q4	10191	783	740	740
q5	7524	2675	2604	2604
q6	217	134	132	132
q7	985	610	568	568
q8	9218	2075	2031	2031
q9	7170	6600	6562	6562
q10	8600	3545	3606	3545
q11	441	233	229	229
q12	466	223	234	223
q13	17759	2974	2994	2974
q14	265	231	230	230
q15	522	486	504	486
q16	534	392	396	392
q17	958	695	709	695
q18	7371	6769	6749	6749
q19	6723	1512	1476	1476
q20	647	308	301	301
q21	3630	2740	2973	2740
q22	367	303	325	303
Total cold run time: 113670 ms
Total hot run time: 38478 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4432	4307	4282	4282
q2	370	277	255	255
q3	2976	2901	2783	2783
q4	1959	1668	1671	1668
q5	5426	5478	5502	5478
q6	221	123	124	123
q7	2295	1994	1988	1988
q8	3265	3426	3421	3421
q9	8768	8702	8912	8702
q10	4105	3929	3939	3929
q11	617	499	505	499
q12	812	663	639	639
q13	16183	3177	3203	3177
q14	315	283	295	283
q15	507	471	482	471
q16	531	431	429	429
q17	1844	1546	1505	1505
q18	8064	7936	7763	7763
q19	1701	1532	1554	1532
q20	2068	1865	1849	1849
q21	9251	5068	4911	4911
q22	557	484	483	483
Total cold run time: 76267 ms
Total hot run time: 56170 ms

doris-robot · 2024-04-19T09:46:28Z

TPC-DS: Total hot run time: 185642 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0079bd0866030c6656cf4b21bcb757949210fb13, data reload: false
query1	895	382	365	365
query2	6216	2731	2385	2385
query3	6665	207	201	201
query4	22714	21343	21353	21343
query5	4119	398	408	398
query6	272	188	172	172
query7	4584	297	274	274
query8	252	182	181	181
query9	8444	2308	2269	2269
query10	430	241	239	239
query11	14672	14177	14223	14177
query12	133	87	82	82
query13	1646	359	361	359
query14	9817	7839	7621	7621
query15	261	182	183	182
query16	8171	265	260	260
query17	1925	576	568	568
query18	2103	282	276	276
query19	283	156	153	153
query20	90	88	83	83
query21	207	123	121	121
query22	5035	4836	4825	4825
query23	33667	33230	33475	33230
query24	10949	3147	3142	3142
query25	626	404	408	404
query26	1162	166	155	155
query27	2471	379	380	379
query28	6986	2051	2078	2051
query29	896	643	667	643
query30	298	189	177	177
query31	1014	767	759	759
query32	89	50	53	50
query33	755	244	253	244
query34	1154	496	500	496
query35	891	695	726	695
query36	1063	912	932	912
query37	143	73	70	70
query38	3429	3401	3369	3369
query39	1653	1601	1558	1558
query40	172	123	128	123
query41	43	43	40	40
query42	101	102	94	94
query43	585	571	558	558
query44	1213	748	727	727
query45	282	259	266	259
query46	1126	760	759	759
query47	2043	1921	1962	1921
query48	387	294	291	291
query49	925	410	443	410
query50	774	387	394	387
query51	6961	6762	6727	6727
query52	96	86	83	83
query53	350	270	270	270
query54	300	221	230	221
query55	77	70	70	70
query56	232	227	213	213
query57	1236	1137	1103	1103
query58	226	202	199	199
query59	3338	3293	3103	3103
query60	253	227	224	224
query61	88	85	85	85
query62	617	445	426	426
query63	296	278	273	273
query64	5034	4034	4028	4028
query65	3062	3017	3029	3017
query66	755	332	328	328
query67	15530	15226	14979	14979
query68	5294	530	586	530
query69	497	300	293	293
query70	1232	1147	1172	1147
query71	1406	1258	1264	1258
query72	6364	2615	2428	2428
query73	715	314	314	314
query74	6775	6436	6254	6254
query75	3427	2603	2647	2603
query76	3439	1064	995	995
query77	575	265	260	260
query78	10883	10316	10231	10231
query79	4826	521	528	521
query80	1875	432	437	432
query81	564	236	252	236
query82	786	95	91	91
query83	310	171	163	163
query84	265	87	81	81
query85	1754	264	259	259
query86	501	295	278	278
query87	3571	3256	3252	3252
query88	4899	2329	2327	2327
query89	478	382	379	379
query90	1974	187	178	178
query91	120	96	94	94
query92	57	44	46	44
query93	5016	521	507	507
query94	1060	178	175	175
query95	387	297	311	297
query96	605	272	262	262
query97	3153	2917	2943	2917
query98	227	212	211	211
query99	1238	856	839	839
Total cold run time: 286555 ms
Total hot run time: 185642 ms

doris-robot · 2024-04-19T09:51:59Z

ClickBench: Total hot run time: 30.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0079bd0866030c6656cf4b21bcb757949210fb13, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.06
query4	1.67	0.08	0.07
query5	0.49	0.49	0.47
query6	1.47	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.53	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.75	0.78	0.77
query15	0.82	0.81	0.79
query16	0.36	0.36	0.37
query17	0.96	0.94	0.99
query18	0.22	0.22	0.23
query19	1.75	1.67	1.68
query20	0.01	0.00	0.01
query21	15.40	0.67	0.65
query22	4.19	7.40	1.87
query23	18.31	1.33	1.27
query24	1.98	0.22	0.21
query25	0.14	0.09	0.08
query26	0.26	0.18	0.17
query27	0.08	0.07	0.08
query28	13.32	1.00	0.97
query29	12.58	3.24	3.23
query30	0.26	0.06	0.06
query31	2.85	0.38	0.38
query32	3.29	0.46	0.46
query33	2.85	2.85	2.78
query34	17.15	4.39	4.42
query35	4.50	4.55	4.46
query36	0.65	0.45	0.46
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.05	0.03
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.45 s
Total hot run time: 30.11 s

doris-robot · 2024-04-19T09:55:05Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0079bd0866030c6656cf4b21bcb757949210fb13 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.8 seconds inserted 10000000 Rows, about 724K ops/s

HappenLee

LGTM

github-actions · 2024-04-19T11:27:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-19T11:27:58Z

PR approved by anyone and no changes requested.

…t the same time (#33736)

…e same time (#39580) ## Proposed changes before we forbid it in #33736, now it's time to let it go. now we support: ```sql mysql> create table auto_dynamic( -> k0 datetime(6) NOT NULL -> ) -> auto partition by range (date_trunc(k0, 'year')) -> ( -> ) -> DISTRIBUTED BY HASH(`k0`) BUCKETS 2 -> properties( -> "dynamic_partition.enable" = "true", -> "dynamic_partition.prefix" = "p", -> "dynamic_partition.start" = "-50", -> "dynamic_partition.end" = "0", -> "dynamic_partition.time_unit" = "year", -> "replication_num" = "1" -> ); Query OK, 0 rows affected (0.12 sec) ``` with this, the users could manage their data more flexible we will refactor the docs of it, and the only recommended way it set end to zero.

…e same time (apache#39580) before we forbid it in apache#33736, now it's time to let it go. now we support: ```sql mysql> create table auto_dynamic( -> k0 datetime(6) NOT NULL -> ) -> auto partition by range (date_trunc(k0, 'year')) -> ( -> ) -> DISTRIBUTED BY HASH(`k0`) BUCKETS 2 -> properties( -> "dynamic_partition.enable" = "true", -> "dynamic_partition.prefix" = "p", -> "dynamic_partition.start" = "-50", -> "dynamic_partition.end" = "0", -> "dynamic_partition.time_unit" = "year", -> "replication_num" = "1" -> ); Query OK, 0 rows affected (0.12 sec) ``` with this, the users could manage their data more flexible we will refactor the docs of it, and the only recommended way it set end to zero.

…e same time (#39580) ## Proposed changes before we forbid it in #33736, now it's time to let it go. now we support: ```sql mysql> create table auto_dynamic( -> k0 datetime(6) NOT NULL -> ) -> auto partition by range (date_trunc(k0, 'year')) -> ( -> ) -> DISTRIBUTED BY HASH(`k0`) BUCKETS 2 -> properties( -> "dynamic_partition.enable" = "true", -> "dynamic_partition.prefix" = "p", -> "dynamic_partition.start" = "-50", -> "dynamic_partition.end" = "0", -> "dynamic_partition.time_unit" = "year", -> "replication_num" = "1" -> ); Query OK, 0 rows affected (0.12 sec) ``` with this, the users could manage their data more flexible we will refactor the docs of it, and the only recommended way it set end to zero.

…t the same time (apache#33736)

…e same time (apache#39580) ## Proposed changes before we forbid it in apache#33736, now it's time to let it go. now we support: ```sql mysql> create table auto_dynamic( -> k0 datetime(6) NOT NULL -> ) -> auto partition by range (date_trunc(k0, 'year')) -> ( -> ) -> DISTRIBUTED BY HASH(`k0`) BUCKETS 2 -> properties( -> "dynamic_partition.enable" = "true", -> "dynamic_partition.prefix" = "p", -> "dynamic_partition.start" = "-50", -> "dynamic_partition.end" = "0", -> "dynamic_partition.time_unit" = "year", -> "replication_num" = "1" -> ); Query OK, 0 rows affected (0.12 sec) ``` with this, the users could manage their data more flexible we will refactor the docs of it, and the only recommended way it set end to zero.

…3778) ## Summary The zh \`release-2.1.2\` page lists four entries under \"Behavior Changed\", the EN page only two. The two missing items are about Auto Partition and were already documented in zh: - **Auto Partition syntax change** (apache/doris#32737) — link to the auto-partitioning doc for the new syntax. - **Auto Partition is no longer allowed to coexist with Dynamic Partition** on the same table (apache/doris#33736). Both are real upstream changes that landed in 2.1.2; port them over to EN so the two release notes are in parity. ## Test plan - [x] EN and zh now list the same four Behavior Changed items, in the same order - [x] PR-ref set (unique) on EN matches what zh references --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

zclllyybb mentioned this pull request Apr 17, 2024
[Doc](auto-partition) Forbid use Auto and Dynamic partition at the same time apache/doris-website#563
Merged

HappenLee reviewed Apr 17, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java

all
0079bd0

zclllyybb force-pushed the fix_partition branch from 9c4b9e0 to 0079bd0Compare April 19, 2024 08:54

BiteTheDDDDt added the dev/2.1.x label Apr 19, 2024

HappenLee approved these changes Apr 19, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 19, 2024

BiteTheDDDDt approved these changes Apr 19, 2024
View reviewed changes

HappenLee merged commit b67ae87 into apache:masterApr 19, 2024

yiguolei pushed a commit that referenced this pull request Apr 19, 2024
[enhancement](auto-partition) Forbid use Auto and Dynamic partition a…
42e9114
…t the same time (#33736)

zclllyybb deleted the fix_partition branch April 19, 2024 17:55

Gabriel39 added dev/2.1.3-merged and removed dev/2.1.x labels Jun 18, 2024

zclllyybb mentioned this pull request Aug 19, 2024
[Enhancement](partition) Support use Auto and Dynamic partition at the same time #39580
Merged

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[enhancement](auto-partition) Forbid use Auto and Dynamic partition a…
8700762
…t the same time (apache#33736)

boluor mentioned this pull request May 25, 2026
[doc] release-2.1.2 EN: backfill two Auto Partition behavior changes apache/doris-website#3778
Merged
2 tasks

Navigation Menu

Uh oh!

[enhancement](auto-partition) Forbid use Auto and Dynamic partition at the same time - #33736

[enhancement](auto-partition) Forbid use Auto and Dynamic partition at the same time#33736
HappenLee merged 1 commit into
apache:masterfrom
zclllyybb:fix_partition

zclllyybb commented Apr 16, 2024•
edited
Loading

Uh oh!

doris-robot commented Apr 16, 2024

Uh oh!

zclllyybb commented Apr 16, 2024

Uh oh!

zclllyybb commented Apr 16, 2024

Uh oh!

Uh oh!

zclllyybb commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

HappenLee left a comment

Uh oh!

github-actionsBot commented Apr 19, 2024

Uh oh!

github-actionsBot commented Apr 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

zclllyybb commented Apr 16, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 16, 2024

Uh oh!

zclllyybb commented Apr 16, 2024

Uh oh!

zclllyybb commented Apr 16, 2024

Uh oh!

Uh oh!

zclllyybb commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 19, 2024

Uh oh!

github-actionsBot commented Apr 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zclllyybb commented Apr 16, 2024•
edited
Loading