Skip to content

[Fix](auto-partition) fix auto partition expr change unexpected - #36345

Merged
zhangstar333 merged 2 commits into
apache:masterfrom
zclllyybb:debug_ap
Jun 18, 2024
Merged

[Fix](auto-partition) fix auto partition expr change unexpected#36345
zhangstar333 merged 2 commits into
apache:masterfrom
zclllyybb:debug_ap

Conversation

@zclllyybb

@zclllyybbzclllyybb commented Jun 15, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

there's a caller used getPartitionExprs and modified the result. the result is a reference to a part of metadata. If when we create the table, the number of editlog didn't reach the limit to generate image file, when it reaches, the image will get the wrong metadata of partitionExprs from memory which was modified.

when Nereids timeout, we will get into trouble in legacy planner here:

privatevoidanalyzerPartitionExpr(Analyzeranalyzer, PartitionInfopartitionInfo) throwsAnalysisException {
ArrayList<Expr> exprs = partitionInfo.getPartitionExprs();
for (Expre : exprs) {
LOG.info("beforeB: {}", e.toSql());
e.analyze(analyzer);
LOG.info("afterB: {}", e.toSql());
}
}
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1561] beforeB: date_trunc(Carding_Day, 'MONTH')
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1563] afterB: date_trunc(`a`.null, 'MONTH')

@doris-robot

doris-robot commented Jun 15, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb

zclllyybb commented Jun 15, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 15, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39786 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a583ded9aba9ed193a109045d9291f50ab7fa12b, data reload: false
------ Round 1 ----------------------------------
q1	17609	4281	4280	4280
q2	2024	199	188	188
q3	10453	1158	1154	1154
q4	10186	794	787	787
q5	7458	2682	2606	2606
q6	215	136	132	132
q7	969	597	593	593
q8	9223	2061	2050	2050
q9	8753	6476	6467	6467
q10	8931	3760	3754	3754
q11	445	237	232	232
q12	469	231	228	228
q13	17764	2987	2960	2960
q14	255	226	220	220
q15	522	466	476	466
q16	503	370	374	370
q17	951	662	626	626
q18	7972	7411	7384	7384
q19	3594	1474	1539	1474
q20	646	313	333	313
q21	4849	3161	3989	3161
q22	403	350	341	341
Total cold run time: 114194 ms
Total hot run time: 39786 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4381	4243	4245	4243
q2	378	256	267	256
q3	2996	2728	2991	2728
q4	1985	1721	1642	1642
q5	5540	5488	5501	5488
q6	225	126	135	126
q7	2197	1844	1910	1844
q8	3263	3389	3369	3369
q9	8702	8623	8788	8623
q10	4058	3842	3786	3786
q11	597	497	483	483
q12	762	634	660	634
q13	16058	3177	3200	3177
q14	308	281	277	277
q15	524	476	480	476
q16	504	439	445	439
q17	1794	1496	1478	1478
q18	8118	8056	7709	7709
q19	1807	1607	1430	1430
q20	2141	1897	1849	1849
q21	9223	4961	4842	4842
q22	613	551	548	548
Total cold run time: 76174 ms
Total hot run time: 55447 ms

@doris-robot

doris-robot commented Jun 15, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 173315 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a583ded9aba9ed193a109045d9291f50ab7fa12b, data reload: false
query1	948	394	371	371
query2	6462	2574	2373	2373
query3	6639	205	208	205
query4	19297	17391	17272	17272
query5	3603	506	460	460
query6	238	156	162	156
query7	4583	299	292	292
query8	353	293	304	293
query9	8555	2390	2357	2357
query10	568	313	296	296
query11	10559	10144	10055	10055
query12	121	97	84	84
query13	1641	367	368	367
query14	10433	7680	6825	6825
query15	232	183	191	183
query16	7721	266	265	265
query17	1921	532	507	507
query18	1976	270	269	269
query19	198	151	149	149
query20	86	81	81	81
query21	204	131	125	125
query22	4392	4080	3990	3990
query23	33737	33633	33650	33633
query24	10760	2897	2841	2841
query25	594	366	376	366
query26	703	160	156	156
query27	2331	332	343	332
query28	5999	2105	2116	2105
query29	866	660	610	610
query30	256	158	159	158
query31	974	766	761	761
query32	90	54	55	54
query33	666	308	289	289
query34	896	506	498	498
query35	743	675	649	649
query36	1126	978	972	972
query37	150	78	69	69
query38	2907	2829	2807	2807
query39	919	858	822	822
query40	211	127	132	127
query41	56	56	74	56
query42	113	102	102	102
query43	632	578	544	544
query44	1066	712	733	712
query45	188	160	165	160
query46	1066	695	727	695
query47	1884	1769	1789	1769
query48	364	294	298	294
query49	839	400	401	400
query50	756	388	398	388
query51	6809	6667	6651	6651
query52	102	97	97	97
query53	366	297	286	286
query54	874	454	445	445
query55	74	74	72	72
query56	285	255	262	255
query57	1112	1065	1062	1062
query58	265	246	244	244
query59	3522	3208	3358	3208
query60	292	270	282	270
query61	114	110	125	110
query62	586	466	458	458
query63	325	291	294	291
query64	8459	2205	1719	1719
query65	3489	3093	3104	3093
query66	739	316	321	316
query67	15244	15161	14883	14883
query68	4562	538	542	538
query69	640	481	413	413
query70	1163	1215	1143	1143
query71	423	277	275	275
query72	7114	5188	5322	5188
query73	753	322	323	322
query74	5940	5567	5520	5520
query75	3430	2715	2669	2669
query76	2613	943	933	933
query77	601	298	303	298
query78	10289	9825	9715	9715
query79	1883	514	518	514
query80	1185	475	449	449
query81	568	221	229	221
query82	768	106	102	102
query83	192	161	161	161
query84	268	85	87	85
query85	1226	301	271	271
query86	446	322	310	310
query87	3263	3069	3038	3038
query88	3872	2340	2315	2315
query89	456	386	377	377
query90	1699	198	192	192
query91	185	102	97	97
query92	60	52	48	48
query93	2053	509	505	505
query94	1047	189	185	185
query95	411	309	316	309
query96	588	263	267	263
query97	3215	3011	3060	3011
query98	213	195	191	191
query99	1170	853	865	853
Total cold run time: 266575 ms
Total hot run time: 173315 ms

@doris-robot

doris-robot commented Jun 15, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.19 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a583ded9aba9ed193a109045d9291f50ab7fa12b, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.06	0.06
query4	1.67	0.07	0.07
query5	0.48	0.48	0.52
query6	1.12	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.54	0.49	0.50
query10	0.55	0.56	0.55
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.59	0.58	0.60
query14	0.78	0.79	0.78
query15	0.85	0.82	0.83
query16	0.35	0.37	0.37
query17	0.98	0.99	1.01
query18	0.23	0.26	0.24
query19	1.83	1.84	1.81
query20	0.02	0.01	0.01
query21	15.41	0.67	0.65
query22	3.93	6.99	2.28
query23	18.27	1.32	1.24
query24	2.18	0.21	0.22
query25	0.15	0.08	0.10
query26	0.25	0.17	0.17
query27	0.08	0.09	0.09
query28	13.23	1.02	1.00
query29	12.61	3.32	3.31
query30	0.26	0.07	0.06
query31	2.84	0.39	0.40
query32	3.25	0.47	0.47
query33	2.92	2.91	2.96
query34	17.09	4.40	4.42
query35	4.55	4.56	4.50
query36	0.65	0.48	0.48
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.17	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.13 s
Total hot run time: 31.19 s

@zclllyybb
zclllyybb marked this pull request as draft June 17, 2024 03:30
@zclllyybb

zclllyybb commented Jun 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@zclllyybbzclllyybb changed the title [Debug](auto-partition) add debug log in binlog for auto partition[Debug](auto-partition) add debug log for auto partitionJun 17, 2024
@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40733 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79d9c3e62c12dd3eff3b2b859df6ccfbb1359316, data reload: false
------ Round 1 ----------------------------------
q1	17878	4562	4423	4423
q2	3000	202	214	202
q3	10790	1170	1193	1170
q4	10761	900	738	738
q5	7535	2709	2673	2673
q6	233	142	139	139
q7	975	617	608	608
q8	9243	2073	2079	2073
q9	8969	6488	6446	6446
q10	8967	3723	3726	3723
q11	447	243	238	238
q12	431	230	230	230
q13	17765	2976	3016	2976
q14	257	233	219	219
q15	507	487	477	477
q16	518	377	379	377
q17	954	733	667	667
q18	7971	7526	7440	7440
q19	6826	1549	1500	1500
q20	684	319	330	319
q21	4918	3760	3916	3760
q22	398	335	337	335
Total cold run time: 120027 ms
Total hot run time: 40733 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4346	4218	4273	4218
q2	369	277	270	270
q3	2993	2694	2785	2694
q4	1883	1625	1654	1625
q5	5277	5265	5278	5265
q6	220	131	132	131
q7	2100	1738	1713	1713
q8	3182	3329	3287	3287
q9	8326	8355	8369	8355
q10	3934	3762	3709	3709
q11	586	521	486	486
q12	778	597	570	570
q13	16321	2996	3004	2996
q14	293	267	268	267
q15	513	484	485	484
q16	458	418	425	418
q17	1781	1487	1477	1477
q18	7571	7563	7385	7385
q19	1978	1467	1482	1467
q20	1985	1777	1771	1771
q21	4863	4822	4806	4806
q22	635	527	553	527
Total cold run time: 70392 ms
Total hot run time: 53921 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 171740 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79d9c3e62c12dd3eff3b2b859df6ccfbb1359316, data reload: false
query1	918	380	374	374
query2	6461	2407	2208	2208
query3	6643	210	207	207
query4	19170	17195	17568	17195
query5	4131	479	497	479
query6	239	164	157	157
query7	4590	293	290	290
query8	299	283	282	282
query9	8456	2380	2386	2380
query10	613	304	282	282
query11	10543	10091	9946	9946
query12	135	88	83	83
query13	1653	365	360	360
query14	10061	6800	7681	6800
query15	211	180	191	180
query16	7829	278	267	267
query17	1830	531	556	531
query18	1963	289	285	285
query19	198	160	172	160
query20	96	88	87	87
query21	209	133	127	127
query22	4258	3981	3882	3882
query23	33607	33335	33046	33046
query24	11860	2876	2921	2876
query25	638	349	364	349
query26	1673	150	149	149
query27	2866	329	322	322
query28	7508	2029	2042	2029
query29	1036	644	588	588
query30	282	149	155	149
query31	941	718	747	718
query32	96	53	58	53
query33	761	275	284	275
query34	965	460	452	452
query35	744	610	621	610
query36	1100	930	944	930
query37	169	69	68	68
query38	2892	2767	2759	2759
query39	858	788	795	788
query40	284	124	128	124
query41	57	53	53	53
query42	122	101	97	97
query43	569	542	523	523
query44	1228	722	731	722
query45	198	166	166	166
query46	1086	737	700	700
query47	1867	1828	1808	1808
query48	377	296	296	296
query49	1165	421	399	399
query50	773	384	394	384
query51	6814	6731	6639	6639
query52	101	94	92	92
query53	352	288	289	288
query54	1031	447	445	445
query55	74	75	76	75
query56	284	254	260	254
query57	1124	1065	1039	1039
query58	253	231	247	231
query59	3479	3055	3035	3035
query60	292	266	302	266
query61	97	90	93	90
query62	660	471	463	463
query63	349	283	288	283
query64	9901	2264	1729	1729
query65	3167	3083	3126	3083
query66	1390	344	348	344
query67	15489	14977	14930	14930
query68	4778	543	534	534
query69	604	416	393	393
query70	1128	1037	1131	1037
query71	410	280	280	280
query72	7167	5178	5485	5178
query73	771	326	331	326
query74	6023	5459	5481	5459
query75	3525	2691	2661	2661
query76	3372	971	914	914
query77	645	318	320	318
query78	10305	9885	9745	9745
query79	2170	512	513	512
query80	1719	480	461	461
query81	588	216	224	216
query82	1047	105	103	103
query83	247	175	188	175
query84	252	91	84	84
query85	1388	292	281	281
query86	478	328	288	288
query87	3314	3071	3120	3071
query88	3480	2342	2364	2342
query89	482	376	376	376
query90	1823	187	195	187
query91	131	101	170	101
query92	66	48	46	46
query93	1978	526	510	510
query94	1178	194	187	187
query95	404	325	311	311
query96	581	267	263	263
query97	3245	3073	3077	3073
query98	232	204	197	197
query99	1159	830	850	830
Total cold run time: 275775 ms
Total hot run time: 171740 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79d9c3e62c12dd3eff3b2b859df6ccfbb1359316, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.06
query5	0.53	0.49	0.49
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.05	0.04
query9	0.54	0.51	0.48
query10	0.54	0.56	0.55
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.78	0.78	0.80
query15	0.86	0.82	0.81
query16	0.39	0.36	0.37
query17	1.03	1.04	1.01
query18	0.21	0.24	0.24
query19	1.79	1.71	1.80
query20	0.01	0.01	0.01
query21	15.44	0.65	0.64
query22	4.02	8.01	1.82
query23	18.25	1.34	1.35
query24	2.05	0.24	0.21
query25	0.15	0.09	0.08
query26	0.26	0.17	0.18
query27	0.08	0.07	0.08
query28	13.24	1.01	1.00
query29	12.59	3.27	3.27
query30	0.26	0.06	0.06
query31	2.84	0.40	0.39
query32	3.26	0.47	0.48
query33	2.86	2.90	2.90
query34	17.10	4.54	4.43
query35	4.49	4.53	4.47
query36	0.66	0.47	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.09 s
Total hot run time: 30.61 s

@zclllyybbzclllyybb changed the title [Debug](auto-partition) add debug log for auto partition[Fix](auto-partition) fix auto partition expr change unexpectedJun 17, 2024
@zclllyybb
zclllyybb marked this pull request as ready for review June 17, 2024 07:19
@zclllyybb

zclllyybb commented Jun 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39443 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e70d1599cea09bd75257a45a4e14dd67cd492a9b, data reload: false
------ Round 1 ----------------------------------
q1	17609	4343	4255	4255
q2	2029	202	195	195
q3	10459	1079	1218	1079
q4	10200	919	771	771
q5	7543	2670	2649	2649
q6	215	132	130	130
q7	949	604	589	589
q8	9206	2066	2032	2032
q9	8922	6441	6429	6429
q10	8978	3739	3684	3684
q11	447	238	234	234
q12	428	235	226	226
q13	17778	2949	2955	2949
q14	267	220	239	220
q15	512	458	462	458
q16	538	373	369	369
q17	954	618	724	618
q18	7972	7317	7345	7317
q19	5032	1383	1498	1383
q20	650	301	324	301
q21	4807	3217	3954	3217
q22	401	338	343	338
Total cold run time: 115896 ms
Total hot run time: 39443 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4382	4242	4224	4224
q2	376	265	285	265
q3	2999	2803	2863	2803
q4	2001	1728	1672	1672
q5	5580	5525	5510	5510
q6	227	128	129	128
q7	2147	1860	1839	1839
q8	3239	3389	3352	3352
q9	8740	8678	8857	8678
q10	4020	3975	3663	3663
q11	566	497	486	486
q12	817	643	640	640
q13	17225	3120	3139	3120
q14	302	278	270	270
q15	527	477	481	477
q16	502	431	435	431
q17	1802	1499	1489	1489
q18	7998	7847	7680	7680
q19	1817	1653	1669	1653
q20	3057	1948	1855	1855
q21	5120	4986	4776	4776
q22	830	558	550	550
Total cold run time: 74274 ms
Total hot run time: 55561 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 173128 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e70d1599cea09bd75257a45a4e14dd67cd492a9b, data reload: false
query1	929	377	379	377
query2	6474	2342	2329	2329
query3	6641	206	203	203
query4	19349	17438	17187	17187
query5	3595	474	473	473
query6	239	158	159	158
query7	4582	291	287	287
query8	300	295	291	291
query9	8401	2394	2386	2386
query10	572	311	280	280
query11	10470	10149	9995	9995
query12	118	91	88	88
query13	1651	373	368	368
query14	9379	6189	6261	6189
query15	223	197	192	192
query16	7143	277	278	277
query17	1010	527	559	527
query18	1951	281	279	279
query19	209	155	161	155
query20	89	82	83	82
query21	206	135	131	131
query22	4803	4487	4452	4452
query23	34167	33558	33689	33558
query24	11312	2829	2825	2825
query25	612	357	361	357
query26	1359	157	154	154
query27	3092	310	322	310
query28	7609	2030	2009	2009
query29	923	611	605	605
query30	228	149	149	149
query31	949	750	735	735
query32	90	52	53	52
query33	732	285	275	275
query34	903	479	475	475
query35	718	606	600	600
query36	1070	920	942	920
query37	167	69	68	68
query38	2873	2775	2726	2726
query39	858	807	819	807
query40	209	131	122	122
query41	54	55	50	50
query42	121	98	97	97
query43	592	558	558	558
query44	1154	718	709	709
query45	194	164	164	164
query46	1085	738	704	704
query47	1856	1805	1762	1762
query48	360	293	310	293
query49	864	425	436	425
query50	780	389	387	387
query51	6826	6636	6642	6636
query52	110	89	102	89
query53	362	295	286	286
query54	859	451	445	445
query55	75	76	76	76
query56	306	278	286	278
query57	1111	1056	1086	1056
query58	264	278	273	273
query59	3338	3162	3134	3134
query60	308	286	287	286
query61	117	112	111	111
query62	612	439	470	439
query63	329	296	299	296
query64	9005	2264	1748	1748
query65	3298	3099	3071	3071
query66	1416	322	340	322
query67	15382	15181	15011	15011
query68	5660	544	530	530
query69	670	426	343	343
query70	1122	1138	1181	1138
query71	459	274	273	273
query72	8244	5866	5703	5703
query73	780	319	322	319
query74	5865	5576	5459	5459
query75	3783	2632	2660	2632
query76	3278	962	898	898
query77	610	305	295	295
query78	10371	9907	9801	9801
query79	3782	514	524	514
query80	1326	475	472	472
query81	546	228	218	218
query82	777	111	103	103
query83	201	167	162	162
query84	277	92	85	85
query85	1472	277	323	277
query86	455	295	336	295
query87	3250	3063	3142	3063
query88	3917	2351	2348	2348
query89	497	383	397	383
query90	1911	197	188	188
query91	128	96	98	96
query92	65	50	50	50
query93	4633	513	509	509
query94	1227	188	185	185
query95	409	316	314	314
query96	594	262	263	262
query97	3242	3038	3059	3038
query98	233	199	199	199
query99	1187	817	842	817
Total cold run time: 277954 ms
Total hot run time: 173128 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e70d1599cea09bd75257a45a4e14dd67cd492a9b, data reload: false
query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.10	0.10
query5	0.50	0.50	0.50
query6	1.14	0.73	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.54	0.48	0.49
query10	0.54	0.56	0.54
query11	0.14	0.11	0.12
query12	0.15	0.12	0.11
query13	0.59	0.59	0.60
query14	0.78	0.79	0.78
query15	0.83	0.82	0.81
query16	0.36	0.35	0.36
query17	0.99	0.98	0.95
query18	0.21	0.24	0.23
query19	1.77	1.73	1.68
query20	0.02	0.01	0.01
query21	15.41	0.65	0.66
query22	4.67	7.24	1.68
query23	18.29	1.48	1.31
query24	2.13	0.22	0.21
query25	0.15	0.09	0.09
query26	0.27	0.17	0.19
query27	0.09	0.08	0.08
query28	13.24	1.01	1.00
query29	12.60	3.22	3.24
query30	0.26	0.07	0.05
query31	2.86	0.39	0.38
query32	3.29	0.47	0.47
query33	2.88	2.95	2.90
query34	17.00	4.67	4.59
query35	4.73	4.59	4.53
query36	0.64	0.47	0.48
query37	0.18	0.16	0.15
query38	0.15	0.15	0.13
query39	0.05	0.04	0.03
query40	0.16	0.15	0.14
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.91 s
Total hot run time: 30.52 s

@zclllyybb

zclllyybb commented Jun 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39495 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b82aeef696c070396f3e1d657fdf60266d1fb6b2, data reload: false
------ Round 1 ----------------------------------
q1	17634	4362	4262	4262
q2	2035	188	183	183
q3	10545	1095	1150	1095
q4	10191	833	833	833
q5	7548	2698	2554	2554
q6	222	139	137	137
q7	990	619	619	619
q8	9660	2073	2041	2041
q9	8855	6438	6472	6438
q10	8877	3713	3691	3691
q11	438	231	232	231
q12	422	224	228	224
q13	17939	2986	2976	2976
q14	271	220	211	211
q15	520	476	486	476
q16	498	380	390	380
q17	961	698	679	679
q18	8097	7567	7262	7262
q19	7340	1400	1574	1400
q20	655	317	333	317
q21	4885	3153	3236	3153
q22	382	333	347	333
Total cold run time: 118965 ms
Total hot run time: 39495 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4410	4265	4250	4250
q2	368	267	255	255
q3	2928	2702	2738	2702
q4	1825	1599	1626	1599
q5	5239	5245	5271	5245
q6	212	123	124	123
q7	2121	1720	1741	1720
q8	3176	3305	3317	3305
q9	8247	8311	8317	8311
q10	3843	3606	3640	3606
q11	583	502	477	477
q12	747	592	565	565
q13	17528	2982	2982	2982
q14	291	262	259	259
q15	513	476	491	476
q16	475	414	423	414
q17	1770	1498	1474	1474
q18	7607	7478	7354	7354
q19	1712	1641	1542	1542
q20	1970	1770	1792	1770
q21	4864	4761	4748	4748
q22	624	530	569	530
Total cold run time: 71053 ms
Total hot run time: 53707 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 172730 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b82aeef696c070396f3e1d657fdf60266d1fb6b2, data reload: false
query1	934	377	375	375
query2	6488	2447	2371	2371
query3	6654	203	206	203
query4	21162	17321	17229	17229
query5	4131	503	444	444
query6	249	155	154	154
query7	4602	289	288	288
query8	318	305	294	294
query9	8273	2388	2375	2375
query10	599	326	279	279
query11	10598	9924	10097	9924
query12	129	85	82	82
query13	1642	366	383	366
query14	10439	6750	6750	6750
query15	248	194	186	186
query16	7811	262	275	262
query17	1610	534	524	524
query18	1961	272	265	265
query19	197	148	160	148
query20	92	90	79	79
query21	211	125	131	125
query22	4410	4097	4083	4083
query23	33479	32976	33001	32976
query24	11992	2849	2911	2849
query25	626	359	371	359
query26	1615	149	152	149
query27	3052	315	310	310
query28	7626	2060	2036	2036
query29	1043	655	633	633
query30	274	150	150	150
query31	950	716	739	716
query32	90	52	53	52
query33	768	285	279	279
query34	990	475	457	457
query35	743	596	591	591
query36	1097	899	964	899
query37	159	69	69	69
query38	2879	2719	2716	2716
query39	852	791	796	791
query40	280	124	128	124
query41	56	54	55	54
query42	113	98	98	98
query43	573	520	547	520
query44	1243	722	738	722
query45	192	165	166	165
query46	1075	728	712	712
query47	1856	1770	1794	1770
query48	371	305	304	304
query49	1193	412	414	412
query50	762	390	378	378
query51	6838	6698	6728	6698
query52	103	94	93	93
query53	362	284	287	284
query54	1016	443	451	443
query55	78	77	78	77
query56	290	274	273	273
query57	1167	1051	1079	1051
query58	257	266	261	261
query59	3312	3315	3225	3225
query60	304	294	287	287
query61	115	113	108	108
query62	680	459	466	459
query63	319	281	287	281
query64	10071	2320	1823	1823
query65	3192	3120	3104	3104
query66	1378	368	341	341
query67	15618	15255	14998	14998
query68	4585	564	539	539
query69	549	359	323	323
query70	1163	1146	1156	1146
query71	411	272	270	270
query72	8063	5556	5600	5556
query73	751	322	323	322
query74	5895	5470	5495	5470
query75	3539	2691	2664	2664
query76	2583	919	974	919
query77	453	290	294	290
query78	10489	10002	9814	9814
query79	5082	518	521	518
query80	3207	457	463	457
query81	561	224	225	224
query82	1212	108	97	97
query83	341	169	165	165
query84	270	83	87	83
query85	2239	318	260	260
query86	483	324	311	311
query87	3267	3048	3086	3048
query88	4615	2360	2368	2360
query89	490	405	380	380
query90	1904	185	183	183
query91	130	99	98	98
query92	60	49	49	49
query93	5531	512	504	504
query94	1316	183	183	183
query95	393	308	317	308
query96	610	264	262	262
query97	3263	2998	3025	2998
query98	229	210	201	201
query99	1200	825	842	825
Total cold run time: 288691 ms
Total hot run time: 172730 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b82aeef696c070396f3e1d657fdf60266d1fb6b2, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.08
query5	0.50	0.49	0.48
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.55	0.50	0.49
query10	0.55	0.56	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.78	0.77
query15	0.82	0.81	0.81
query16	0.36	0.36	0.36
query17	0.96	0.99	0.96
query18	0.23	0.24	0.25
query19	1.82	1.71	1.73
query20	0.02	0.01	0.01
query21	15.44	0.65	0.65
query22	4.42	7.38	1.72
query23	18.31	1.35	1.26
query24	2.05	0.22	0.22
query25	0.16	0.09	0.08
query26	0.25	0.18	0.17
query27	0.08	0.08	0.08
query28	13.27	1.01	1.00
query29	12.64	3.31	3.27
query30	0.26	0.07	0.05
query31	2.89	0.38	0.39
query32	3.26	0.47	0.47
query33	2.92	2.92	2.88
query34	17.08	4.44	4.38
query35	4.50	4.51	4.58
query36	0.66	0.46	0.48
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.14	0.17
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.03	0.05
Total cold run time: 109.65 s
Total hot run time: 30.35 s

@zclllyybb

zclllyybb commented Jun 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40502 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1dd0f65ad3010477ac418f365f15e7a4c87f41a9, data reload: false
------ Round 1 ----------------------------------
q1	17631	4328	4252	4252
q2	2025	190	191	190
q3	10458	1144	1093	1093
q4	10182	805	782	782
q5	7493	2644	2650	2644
q6	226	135	136	135
q7	948	613	600	600
q8	9223	2048	2087	2048
q9	8880	6470	6456	6456
q10	9048	3704	3754	3704
q11	467	233	237	233
q12	465	239	222	222
q13	17764	2952	2956	2952
q14	275	229	218	218
q15	512	491	481	481
q16	531	380	385	380
q17	971	760	705	705
q18	7833	7428	7320	7320
q19	4889	1514	1454	1454
q20	657	332	333	332
q21	4900	3968	3962	3962
q22	400	340	339	339
Total cold run time: 115778 ms
Total hot run time: 40502 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4392	4298	4262	4262
q2	381	276	275	275
q3	2968	2942	2869	2869
q4	2019	1737	1695	1695
q5	5685	5543	5519	5519
q6	230	134	131	131
q7	2253	1934	1874	1874
q8	3292	3438	3448	3438
q9	8757	8764	8864	8764
q10	4139	3939	3823	3823
q11	618	499	502	499
q12	842	660	656	656
q13	16092	3421	3370	3370
q14	320	302	292	292
q15	541	501	498	498
q16	534	463	470	463
q17	1849	1546	1529	1529
q18	8546	8378	8088	8088
q19	3373	1688	1682	1682
q20	3066	1877	1889	1877
q21	9590	4938	4958	4938
q22	649	568	548	548
Total cold run time: 80136 ms
Total hot run time: 57090 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 173106 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1dd0f65ad3010477ac418f365f15e7a4c87f41a9, data reload: false
query1	938	372	380	372
query2	6482	2328	2207	2207
query3	6640	203	207	203
query4	19503	17328	17221	17221
query5	3598	480	463	463
query6	239	163	159	159
query7	4594	302	289	289
query8	359	292	293	292
query9	8527	2409	2350	2350
query10	585	308	272	272
query11	10622	10231	9967	9967
query12	118	93	82	82
query13	1641	361	359	359
query14	9806	7839	7687	7687
query15	260	188	191	188
query16	8052	266	268	266
query17	1889	544	512	512
query18	2114	275	274	274
query19	241	153	149	149
query20	91	86	82	82
query21	215	133	127	127
query22	4435	4024	4021	4021
query23	33958	33748	33423	33423
query24	10763	2859	2931	2859
query25	569	381	360	360
query26	706	152	153	152
query27	2241	332	320	320
query28	5970	2099	2081	2081
query29	892	648	623	623
query30	257	152	150	150
query31	966	758	737	737
query32	90	54	52	52
query33	677	292	283	283
query34	902	478	474	474
query35	783	656	626	626
query36	1111	975	977	975
query37	138	72	72	72
query38	3012	2862	2875	2862
query39	927	814	817	814
query40	220	130	139	130
query41	57	52	53	52
query42	108	105	103	103
query43	573	543	535	535
query44	1068	709	719	709
query45	187	155	162	155
query46	1061	752	719	719
query47	1867	1764	1731	1731
query48	383	291	296	291
query49	826	396	405	396
query50	760	425	385	385
query51	6765	6696	6612	6612
query52	103	86	88	86
query53	360	292	294	292
query54	869	440	456	440
query55	73	71	71	71
query56	271	252	260	252
query57	1103	1024	1043	1024
query58	235	265	238	238
query59	3336	3204	3088	3088
query60	322	261	273	261
query61	93	86	91	86
query62	597	446	436	436
query63	312	287	290	287
query64	8609	2214	1721	1721
query65	3180	3081	3111	3081
query66	761	320	326	320
query67	15443	15081	14912	14912
query68	4662	524	537	524
query69	580	409	388	388
query70	1112	1139	1111	1111
query71	446	269	264	264
query72	7558	5117	5700	5117
query73	771	321	328	321
query74	5845	5443	5413	5413
query75	3483	2682	2652	2652
query76	2915	956	944	944
query77	616	294	292	292
query78	10216	9993	9738	9738
query79	2526	507	498	498
query80	1395	460	447	447
query81	541	223	218	218
query82	741	101	98	98
query83	198	164	164	164
query84	274	87	81	81
query85	1241	286	267	267
query86	414	336	329	329
query87	3285	3072	3053	3053
query88	4205	2345	2328	2328
query89	475	392	378	378
query90	1778	184	189	184
query91	123	102	107	102
query92	58	54	58	54
query93	3140	498	488	488
query94	1050	191	197	191
query95	475	308	306	306
query96	594	268	262	262
query97	3264	3039	3099	3039
query98	228	204	206	204
query99	1370	833	860	833
Total cold run time: 270032 ms
Total hot run time: 173106 ms

@doris-robot

doris-robot commented Jun 17, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1dd0f65ad3010477ac418f365f15e7a4c87f41a9, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.49	0.48
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.56	0.49	0.49
query10	0.53	0.56	0.54
query11	0.15	0.11	0.13
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.77	0.78	0.78
query15	0.84	0.82	0.81
query16	0.36	0.36	0.36
query17	0.99	1.02	1.03
query18	0.21	0.26	0.25
query19	1.78	1.73	1.70
query20	0.02	0.01	0.01
query21	15.41	0.67	0.66
query22	5.06	6.54	1.69
query23	18.28	1.37	1.19
query24	2.11	0.24	0.21
query25	0.16	0.08	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.07
query28	13.25	1.01	1.00
query29	12.64	3.23	3.19
query30	0.26	0.06	0.06
query31	2.87	0.38	0.38
query32	3.26	0.48	0.47
query33	2.91	2.85	2.86
query34	17.22	4.71	4.67
query35	4.74	4.65	4.48
query36	0.65	0.46	0.46
query37	0.18	0.16	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.15	0.16
query41	0.08	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.55 s
Total hot run time: 30.46 s

HappenLee
HappenLee approved these changes Jun 18, 2024
View reviewed changes

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2024
@github-actions

github-actionsBot commented Jun 18, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jun 18, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@zhangstar333
zhangstar333 merged commit c0780e6 into apache:masterJun 18, 2024
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Jun 19, 2024
@zclllyybb
[Fix](auto-partition) fix auto partition expr change unexpected (apac…
b1f1d0e
…he#36345)
## Proposed changes
Issue Number: close #xxx
there's a caller used `getPartitionExprs` and modified the result. the
result is a reference to a part of metadata. If when we create the
table, the number of editlog didn't reach the limit to generate image
file, when it reaches, the image will get the wrong metadata of
`partitionExprs` from memory which was modified.
currently we don't know why, but this root cause has been confirmed by:
```java
private void analyzerPartitionExpr(Analyzer analyzer, PartitionInfo partitionInfo) throws AnalysisException {
ArrayList<Expr> exprs = partitionInfo.getPartitionExprs();
for (Expr e : exprs) {
LOG.info("beforeB: {}", e.toSql());
e.analyze(analyzer);
LOG.info("afterB: {}", e.toSql());
}
}
```
```log
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1561] beforeB: date_trunc(Carding_Day, 'MONTH')
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1563] afterB: date_trunc(`a`.null, 'MONTH')
```
@zclllyybbzclllyybb mentioned this pull request Jun 19, 2024
Merged
@zhangstar333zhangstar333 added usercase Important user case type label dev/2.1.x labels Jun 19, 2024
morningman pushed a commit that referenced this pull request Jun 20, 2024
@zclllyybb
[branch-2.1](auto-partition) fix auto partition expr change unexpected (
6df1a9a
#36345) (#36514)
pick #36345
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@zclllyybb@dataroaring
[Fix](auto-partition) fix auto partition expr change unexpected (#36345)
82d09cf
## Proposed changes
Issue Number: close #xxx
there's a caller used `getPartitionExprs` and modified the result. the
result is a reference to a part of metadata. If when we create the
table, the number of editlog didn't reach the limit to generate image
file, when it reaches, the image will get the wrong metadata of
`partitionExprs` from memory which was modified.
currently we don't know why, but this root cause has been confirmed by:
```java
private void analyzerPartitionExpr(Analyzer analyzer, PartitionInfo partitionInfo) throws AnalysisException {
ArrayList<Expr> exprs = partitionInfo.getPartitionExprs();
for (Expr e : exprs) {
LOG.info("beforeB: {}", e.toSql());
e.analyze(analyzer);
LOG.info("afterB: {}", e.toSql());
}
}
```
```log
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1561] beforeB: date_trunc(Carding_Day, 'MONTH')
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1563] afterB: date_trunc(`a`.null, 'MONTH')
```
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@zclllyybb@HappenLee
[Fix](auto-partition) fix auto partition expr change unexpected (apac…
6986cb9
…he#36345)
## Proposed changes
Issue Number: close #xxx
there's a caller used `getPartitionExprs` and modified the result. the
result is a reference to a part of metadata. If when we create the
table, the number of editlog didn't reach the limit to generate image
file, when it reaches, the image will get the wrong metadata of
`partitionExprs` from memory which was modified.
currently we don't know why, but this root cause has been confirmed by:
```java
private void analyzerPartitionExpr(Analyzer analyzer, PartitionInfo partitionInfo) throws AnalysisException {
ArrayList<Expr> exprs = partitionInfo.getPartitionExprs();
for (Expr e : exprs) {
LOG.info("beforeB: {}", e.toSql());
e.analyze(analyzer);
LOG.info("afterB: {}", e.toSql());
}
}
```
```log
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1561] beforeB: date_trunc(Carding_Day, 'MONTH')
2024-06-18 21:49:07,449 INFO (mysql-nio-pool-154|695) [OlapScanNode.analyzerPartitionExpr():1563] afterB: date_trunc(`a`.null, 'MONTH')
```
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HappenLeeHappenLeeHappenLee approved these changes
@zhangstar333zhangstar333zhangstar333 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.4-mergeddev/3.0.0-mergedreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zclllyybb@doris-robot@HappenLee@zhangstar333@morningman@dataroaring