Skip to content

[Feat](nereids) support generated column - #35284

Merged
morrySnow merged 21 commits into
apache:masterfrom
feiniaofeiafei:generated_column
Jun 25, 2024
Merged

[Feat](nereids) support generated column#35284
morrySnow merged 21 commits into
apache:masterfrom
feiniaofeiafei:generated_column

Conversation

@feiniaofeiafei

@feiniaofeiafeifeiniaofeiafei commented May 23, 2024•
edited by morrySnow
Loading

Copy link
Copy Markdown
Collaborator

This pr has added support for generated columns. Values of a generated column are computed from an expression included in the column definition. Here is an example:

mysql> CREATE TABLE products (
product_id INT,
price DECIMAL(10,2),
quantity INT,
total_value DECIMAL(10,2) GENERATED ALWAYS AS (price * quantity)
) DISTRIBUTED BY HASH(product_id) PROPERTIES ("replication_num"="1");
mysql>insert into products values(1, 10.00, 10, default);
mysql>insert into products(product_id, price, quantity) values(1, 20.00, 10);
mysql>select*from products;
+------------+-------+----------+-------------+
| product_id | price | quantity | total_value |
+------------+-------+----------+-------------+
| 1 | 10.00 | 10 | 100.00 |
| 1 | 20.00 | 10 | 200.00 |
+------------+-------+----------+-------------+

related docs in apache/doris-website#715

@doris-robot

doris-robot commented May 23, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@feiniaofeiafei

feiniaofeiafei commented May 23, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented May 23, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 42137 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e19c36c0dff499535ca7be13d161a5251165049, data reload: false
------ Round 1 ----------------------------------
q1	17601	4392	4281	4281
q2	2008	192	183	183
q3	10468	1269	1213	1213
q4	10199	907	761	761
q5	7459	2679	2751	2679
q6	236	135	141	135
q7	965	629	608	608
q8	9223	2099	2084	2084
q9	10163	6711	6686	6686
q10	9224	3945	3948	3945
q11	456	259	251	251
q12	478	238	245	238
q13	17537	3193	3273	3193
q14	257	202	212	202
q15	511	491	468	468
q16	528	407	404	404
q17	1015	758	735	735
q18	8498	7867	7847	7847
q19	3326	1592	1544	1544
q20	666	318	321	318
q21	5359	4085	4189	4085
q22	370	277	301	277
Total cold run time: 116547 ms
Total hot run time: 42137 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4572	4448	4403	4403
q2	382	286	277	277
q3	3212	2954	2834	2834
q4	1913	1699	1731	1699
q5	5500	5497	5500	5497
q6	211	127	129	127
q7	2183	1822	1836	1822
q8	3228	3445	3361	3361
q9	8706	8628	8603	8603
q10	3938	3761	3827	3761
q11	624	503	531	503
q12	793	628	614	614
q13	15922	3190	3244	3190
q14	303	274	279	274
q15	532	483	495	483
q16	510	462	430	430
q17	1812	1486	1431	1431
q18	7732	8395	7526	7526
q19	1690	1601	1537	1537
q20	1991	1805	1787	1787
q21	7916	4829	4741	4741
q22	622	500	522	500
Total cold run time: 74292 ms
Total hot run time: 55400 ms

@doris-robot

doris-robot commented May 23, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 169485 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e19c36c0dff499535ca7be13d161a5251165049, data reload: false
query1	932	378	399	378
query2	6428	2302	2266	2266
query3	6645	203	204	203
query4	19396	17225	17220	17220
query5	4154	418	411	411
query6	251	154	155	154
query7	4590	310	296	296
query8	242	185	180	180
query9	8602	2464	2445	2445
query10	464	299	265	265
query11	10486	10239	9930	9930
query12	133	87	93	87
query13	1637	357	360	357
query14	10171	7441	7583	7441
query15	209	179	179	179
query16	7732	274	260	260
query17	1742	522	524	522
query18	1945	270	272	270
query19	208	155	155	155
query20	95	83	83	83
query21	199	134	150	134
query22	4218	3981	3846	3846
query23	33380	32993	32997	32993
query24	6858	2856	2816	2816
query25	570	366	356	356
query26	713	158	157	157
query27	2031	325	326	325
query28	3544	2078	2081	2078
query29	844	600	609	600
query30	261	171	177	171
query31	940	769	724	724
query32	88	53	55	53
query33	513	280	278	278
query34	863	476	477	476
query35	715	613	609	609
query36	1047	922	898	898
query37	109	75	73	73
query38	2865	2744	2756	2744
query39	891	776	798	776
query40	194	125	125	125
query41	46	44	44	44
query42	102	97	98	97
query43	577	543	551	543
query44	1086	733	744	733
query45	176	158	163	158
query46	1055	699	711	699
query47	1825	1746	1757	1746
query48	377	295	297	295
query49	813	376	387	376
query50	768	393	383	383
query51	6736	6585	6607	6585
query52	102	94	89	89
query53	348	284	285	284
query54	546	438	427	427
query55	75	74	70	70
query56	305	253	248	248
query57	1105	1057	1042	1042
query58	251	212	220	212
query59	3358	3104	3010	3010
query60	276	252	253	252
query61	87	88	86	86
query62	561	465	450	450
query63	311	281	290	281
query64	8447	2223	1689	1689
query65	3196	3101	3135	3101
query66	783	330	320	320
query67	15091	14953	14890	14890
query68	4547	538	548	538
query69	454	264	269	264
query70	1148	1142	1122	1122
query71	391	277	274	274
query72	7448	5384	2716	2716
query73	711	326	325	325
query74	6134	5507	5547	5507
query75	3271	2603	2601	2601
query76	2303	980	1047	980
query77	393	269	267	267
query78	10163	9775	9806	9775
query79	1223	522	521	521
query80	834	452	430	430
query81	540	239	245	239
query82	582	97	94	94
query83	238	193	168	168
query84	249	87	85	85
query85	1127	268	330	268
query86	451	289	284	284
query87	3323	3152	3120	3120
query88	4204	2431	2436	2431
query89	471	387	380	380
query90	2080	193	194	193
query91	123	95	94	94
query92	59	49	49	49
query93	1843	513	493	493
query94	1090	190	191	190
query95	395	305	313	305
query96	585	266	266	266
query97	3176	3007	2997	2997
query98	246	221	218	218
query99	1167	852	847	847
Total cold run time: 256277 ms
Total hot run time: 169485 ms

@doris-robot

doris-robot commented May 23, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e19c36c0dff499535ca7be13d161a5251165049, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.22	0.06	0.05
query4	1.67	0.10	0.10
query5	0.51	0.49	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.50	0.49
query10	0.54	0.55	0.54
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.78	0.79	0.77
query15	0.82	0.80	0.80
query16	0.37	0.38	0.34
query17	0.96	0.97	1.00
query18	0.22	0.24	0.27
query19	1.81	1.70	1.72
query20	0.01	0.01	0.01
query21	15.74	0.66	0.65
query22	4.11	6.88	2.45
query23	18.35	1.37	1.22
query24	1.99	0.22	0.22
query25	0.16	0.08	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.09
query28	13.33	1.00	1.00
query29	13.15	3.35	3.30
query30	0.24	0.06	0.05
query31	2.88	0.40	0.39
query32	3.25	0.47	0.47
query33	2.87	2.91	2.97
query34	17.04	4.49	4.45
query35	4.58	4.49	4.52
query36	0.73	0.49	0.48
query37	0.17	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.04	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.09 s
Total hot run time: 31.16 s

@feiniaofeiafei

feiniaofeiafei commented May 23, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run p0

@morrySnow

morrySnow commented May 24, 2024

Copy link
Copy Markdown
Contributor

Add a description explaining the functionality scope of generated columns, and include simple usage examples. Additionally, update the Doris documentation in https://github.com/apache/doris-website to include a description of this functionality.

@feiniaofeiafei

feiniaofeiafei commented May 27, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

1 similar comment
@feiniaofeiafei

feiniaofeiafei commented May 27, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41819 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6ee3adefe8f5fbf61b2cbdd0a8f7c1c1764fbc3, data reload: false
------ Round 1 ----------------------------------
q1	17629	4396	4275	4275
q2	2005	181	188	181
q3	10483	1179	1216	1179
q4	10203	799	765	765
q5	7471	2698	2794	2698
q6	219	136	135	135
q7	958	615	612	612
q8	9217	2129	2095	2095
q9	9444	6688	6684	6684
q10	9288	4010	3936	3936
q11	436	257	257	257
q12	436	237	232	232
q13	17263	3212	3233	3212
q14	279	236	218	218
q15	501	466	472	466
q16	481	390	400	390
q17	1005	716	658	658
q18	8265	7751	7661	7661
q19	6770	1584	1574	1574
q20	629	323	310	310
q21	5146	4005	4002	4002
q22	342	279	288	279
Total cold run time: 118470 ms
Total hot run time: 41819 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4631	4551	4505	4505
q2	392	273	261	261
q3	3173	2987	2911	2911
q4	1899	1647	1656	1647
q5	5515	5555	5540	5540
q6	221	128	128	128
q7	2176	1830	1874	1830
q8	3286	3393	3430	3393
q9	8733	8754	8677	8677
q10	3974	3943	3860	3860
q11	589	489	494	489
q12	807	628	648	628
q13	17285	3341	3448	3341
q14	319	289	303	289
q15	528	502	486	486
q16	512	454	455	454
q17	1801	1522	1481	1481
q18	8613	8108	7728	7728
q19	1891	1588	1546	1546
q20	2022	1808	1816	1808
q21	11924	4742	4661	4661
q22	591	492	497	492
Total cold run time: 80882 ms
Total hot run time: 56155 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 168894 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6ee3adefe8f5fbf61b2cbdd0a8f7c1c1764fbc3, data reload: false
query1	908	386	385	385
query2	6894	2369	2333	2333
query3	6647	204	209	204
query4	19179	17282	17016	17016
query5	4190	419	425	419
query6	241	158	166	158
query7	4589	306	297	297
query8	233	178	174	174
query9	8487	2348	2330	2330
query10	468	279	269	269
query11	10555	10111	10047	10047
query12	136	92	87	87
query13	1636	362	352	352
query14	10015	7689	7086	7086
query15	198	170	168	168
query16	7756	270	264	264
query17	1768	527	526	526
query18	1950	265	272	265
query19	197	169	153	153
query20	96	82	84	82
query21	193	133	129	129
query22	4098	3935	3865	3865
query23	33318	33103	32972	32972
query24	10893	2987	2827	2827
query25	624	372	382	372
query26	1411	151	156	151
query27	2993	312	320	312
query28	7308	2016	1995	1995
query29	931	613	607	607
query30	230	151	154	151
query31	974	747	751	747
query32	88	52	51	51
query33	753	273	264	264
query34	982	499	482	482
query35	735	596	591	591
query36	1068	901	902	901
query37	132	65	66	65
query38	2904	2781	2748	2748
query39	863	783	787	783
query40	194	125	120	120
query41	47	44	43	43
query42	102	98	97	97
query43	570	565	535	535
query44	1187	732	742	732
query45	179	162	167	162
query46	1072	745	702	702
query47	1895	1782	1768	1768
query48	368	296	291	291
query49	876	387	391	387
query50	768	379	387	379
query51	6898	6750	6790	6750
query52	96	93	87	87
query53	352	281	294	281
query54	849	454	432	432
query55	75	70	74	70
query56	269	255	259	255
query57	1147	1041	1065	1041
query58	240	219	241	219
query59	3460	3031	3303	3031
query60	264	261	254	254
query61	94	87	88	87
query62	607	463	445	445
query63	308	281	276	276
query64	9875	2218	1749	1749
query65	3426	3108	3109	3108
query66	1370	325	336	325
query67	15268	15765	14858	14858
query68	4640	515	524	515
query69	487	271	261	261
query70	1178	1173	1158	1158
query71	426	265	269	265
query72	8309	5593	2700	2700
query73	719	322	322	322
query74	6011	5554	5674	5554
query75	3587	2629	2603	2603
query76	2981	907	962	907
query77	624	262	262	262
query78	10100	9947	9639	9639
query79	2212	517	513	513
query80	1731	438	429	429
query81	509	226	219	219
query82	1210	88	96	88
query83	298	186	170	170
query84	253	89	84	84
query85	1547	282	267	267
query86	467	306	330	306
query87	3280	3082	3093	3082
query88	4067	2429	2431	2429
query89	487	389	387	387
query90	1962	184	185	184
query91	127	95	96	95
query92	58	49	47	47
query93	2081	500	494	494
query94	1262	195	195	195
query95	416	323	386	323
query96	586	270	267	267
query97	3187	2997	2967	2967
query98	241	221	216	216
query99	1212	850	834	834
Total cold run time: 274752 ms
Total hot run time: 168894 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6ee3adefe8f5fbf61b2cbdd0a8f7c1c1764fbc3, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.09
query5	0.50	0.50	0.51
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.53	0.49	0.48
query10	0.51	0.54	0.52
query11	0.16	0.12	0.11
query12	0.14	0.12	0.11
query13	0.59	0.59	0.59
query14	0.77	0.77	0.80
query15	0.82	0.81	0.81
query16	0.38	0.36	0.37
query17	1.01	0.96	1.04
query18	0.23	0.23	0.26
query19	1.78	1.75	1.79
query20	0.01	0.02	0.01
query21	15.43	0.69	0.69
query22	4.94	6.48	1.87
query23	18.28	1.33	1.21
query24	1.84	0.22	0.21
query25	0.14	0.09	0.09
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.26	1.00	0.99
query29	13.20	3.33	3.27
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.31	0.48	0.47
query33	2.86	2.83	2.90
query34	17.28	4.42	4.50
query35	4.50	4.53	4.66
query36	0.65	0.46	0.45
query37	0.17	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.13	0.18
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.03	0.03	0.04
Total cold run time: 110.49 s
Total hot run time: 30.43 s

morrySnow
morrySnow reviewed May 27, 2024
View reviewed changes
Comment threadfe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
Comment threadfe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
Comment threadfe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
Comment thread...-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java Outdated
@feiniaofeiafei

feiniaofeiafei commented May 31, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented May 31, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41123 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19c5b5838569458a54621a2a481d9bd44e6b13a0, data reload: false
------ Round 1 ----------------------------------
q1	17602	4338	4242	4242
q2	2017	196	193	193
q3	10450	1184	1130	1130
q4	10202	806	862	806
q5	7469	2719	2732	2719
q6	223	137	137	137
q7	965	618	622	618
q8	9218	2126	2076	2076
q9	9102	6676	6689	6676
q10	9265	3988	3956	3956
q11	441	250	242	242
q12	487	230	231	230
q13	17253	3200	3246	3200
q14	276	226	228	226
q15	505	467	472	467
q16	462	388	402	388
q17	979	768	656	656
q18	8465	7726	7783	7726
q19	4285	1602	1505	1505
q20	626	329	312	312
q21	5173	4043	3283	3283
q22	395	335	340	335
Total cold run time: 115860 ms
Total hot run time: 41123 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4565	4463	4454	4454
q2	367	271	283	271
q3	3124	2905	2915	2905
q4	1914	1649	1603	1603
q5	5396	5528	5531	5528
q6	221	127	133	127
q7	2244	1755	1801	1755
q8	3239	3405	3345	3345
q9	8520	8650	8643	8643
q10	4067	3792	3656	3656
q11	593	491	503	491
q12	825	631	614	614
q13	15936	3123	3200	3123
q14	293	284	276	276
q15	527	474	480	474
q16	518	441	451	441
q17	1830	1541	1516	1516
q18	8097	7661	7523	7523
q19	1674	1461	1551	1461
q20	2010	1784	1808	1784
q21	12509	4707	4768	4707
q22	585	539	537	537
Total cold run time: 79054 ms
Total hot run time: 55234 ms

@doris-robot

doris-robot commented May 31, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 170158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19c5b5838569458a54621a2a481d9bd44e6b13a0, data reload: false
query1	928	381	380	380
query2	6429	2430	2446	2430
query3	6635	206	202	202
query4	19151	17502	17197	17197
query5	4161	436	431	431
query6	253	155	160	155
query7	4593	307	299	299
query8	343	287	280	280
query9	8497	2382	2399	2382
query10	443	266	261	261
query11	10491	10055	9984	9984
query12	139	90	91	90
query13	1651	376	367	367
query14	10196	7807	7682	7682
query15	236	188	187	187
query16	7794	270	266	266
query17	1697	531	526	526
query18	1960	273	296	273
query19	197	151	151	151
query20	89	83	85	83
query21	194	133	130	130
query22	4095	3883	3885	3883
query23	33656	32959	33196	32959
query24	8960	2886	2865	2865
query25	584	349	355	349
query26	704	167	163	163
query27	2214	333	333	333
query28	5720	2057	2074	2057
query29	868	610	601	601
query30	240	151	154	151
query31	947	766	753	753
query32	93	58	53	53
query33	644	269	270	269
query34	874	486	501	486
query35	724	606	606	606
query36	1073	913	927	913
query37	105	66	67	66
query38	2871	2783	2778	2778
query39	822	807	809	807
query40	200	125	122	122
query41	54	51	50	50
query42	106	98	99	98
query43	589	566	552	552
query44	1059	725	748	725
query45	184	174	172	172
query46	1062	728	742	728
query47	1868	1758	1788	1758
query48	368	286	306	286
query49	828	391	402	391
query50	773	386	394	386
query51	6739	6771	6577	6577
query52	107	96	89	89
query53	359	296	295	295
query54	755	430	455	430
query55	76	75	73	73
query56	261	245	240	240
query57	1080	1030	1010	1010
query58	234	225	207	207
query59	3478	3392	3011	3011
query60	267	262	257	257
query61	96	88	86	86
query62	598	449	448	448
query63	324	296	300	296
query64	8544	2337	1748	1748
query65	3177	3120	3177	3120
query66	782	335	332	332
query67	15251	15202	14907	14907
query68	4514	534	530	530
query69	446	272	269	269
query70	1146	1091	1084	1084
query71	387	272	270	270
query72	7834	2700	2581	2581
query73	709	322	319	319
query74	6047	5714	5635	5635
query75	3369	2645	2620	2620
query76	2499	1029	967	967
query77	417	323	267	267
query78	10291	9756	9803	9756
query79	2447	508	524	508
query80	945	440	436	436
query81	538	221	212	212
query82	949	89	92	89
query83	237	175	174	174
query84	249	84	87	84
query85	1161	277	266	266
query86	436	328	314	314
query87	3327	3129	3154	3129
query88	3802	2363	2359	2359
query89	496	403	395	395
query90	2003	202	193	193
query91	122	154	94	94
query92	58	50	47	47
query93	1798	520	500	500
query94	1260	187	183	183
query95	410	315	320	315
query96	588	267	264	264
query97	3222	3047	3003	3003
query98	254	222	218	218
query99	1155	835	844	835
Total cold run time: 263606 ms
Total hot run time: 170158 ms

@doris-robot

doris-robot commented May 31, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19c5b5838569458a54621a2a481d9bd44e6b13a0, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.04	0.05
query3	0.24	0.06	0.05
query4	1.65	0.09	0.08
query5	0.49	0.48	0.50
query6	1.14	0.72	0.75
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.48	0.50
query10	0.55	0.56	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.60	0.60
query14	0.78	0.77	0.77
query15	0.83	0.81	0.81
query16	0.35	0.37	0.38
query17	0.93	0.97	0.98
query18	0.23	0.24	0.24
query19	1.76	1.81	1.74
query20	0.01	0.01	0.01
query21	15.46	0.70	0.67
query22	4.71	6.30	2.85
query23	18.37	1.40	1.29
query24	1.96	0.23	0.23
query25	0.16	0.09	0.08
query26	0.27	0.16	0.17
query27	0.08	0.08	0.09
query28	13.27	1.05	1.02
query29	13.29	3.27	3.25
query30	0.24	0.05	0.07
query31	2.84	0.38	0.37
query32	3.29	0.46	0.47
query33	2.86	2.87	2.97
query34	17.06	4.44	4.48
query35	4.49	4.48	4.63
query36	0.65	0.46	0.47
query37	0.18	0.16	0.15
query38	0.14	0.14	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.15
query41	0.10	0.04	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.33 s
Total hot run time: 31.56 s

@feiniaofeiafei

feiniaofeiafei commented May 31, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

1 similar comment
@feiniaofeiafei

feiniaofeiafei commented May 31, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@feiniaofeiafei

feiniaofeiafei commented May 31, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented May 31, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39818 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1a52bcaeb985acf402d056bef770c1520b98d2f, data reload: false
------ Round 1 ----------------------------------
q1	17660	4556	4300	4300
q2	2039	197	197	197
q3	10575	3078	1084	1084
q4	10172	689	840	689
q5	7509	2728	2649	2649
q6	218	138	138	138
q7	947	613	602	602
q8	9213	2111	2066	2066
q9	8938	6557	6507	6507
q10	8987	3737	3690	3690
q11	467	248	240	240
q12	533	229	220	220
q13	17769	2974	2978	2974
q14	262	215	227	215
q15	528	494	470	470
q16	525	380	383	380
q17	984	710	725	710
q18	8193	7502	7481	7481
q19	4705	1499	1364	1364
q20	654	318	297	297
q21	5012	3228	3820	3228
q22	378	317	320	317
Total cold run time: 116268 ms
Total hot run time: 39818 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4399	4266	4274	4266
q2	362	261	274	261
q3	2985	2779	2771	2771
q4	1886	1575	1631	1575
q5	5320	5335	5303	5303
q6	219	125	126	125
q7	2097	1735	1721	1721
q8	3222	3331	3329	3329
q9	8468	8419	8371	8371
q10	3895	3711	3670	3670
q11	598	487	489	487
q12	755	602	585	585
q13	16464	3132	3101	3101
q14	333	268	274	268
q15	518	481	484	481
q16	497	445	439	439
q17	1831	1525	1500	1500
q18	8017	7564	7414	7414
q19	1711	1627	1561	1561
q20	2030	1801	1840	1801
q21	4939	4765	4923	4765
q22	593	543	537	537
Total cold run time: 71139 ms
Total hot run time: 54331 ms

@feiniaofeiafei

feiniaofeiafei commented Jun 2, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented Jun 2, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41289 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15c2973da258d3977f10d7fe06448a72fe253953, data reload: false
------ Round 1 ----------------------------------
q1	17635	4387	4354	4354
q2	2026	206	203	203
q3	10425	1152	1224	1152
q4	10228	831	902	831
q5	7478	2759	2832	2759
q6	225	140	141	140
q7	994	637	606	606
q8	9220	2197	2143	2143
q9	9450	6722	6812	6722
q10	9603	3893	3882	3882
q11	461	252	249	249
q12	463	254	240	240
q13	17246	3192	3216	3192
q14	290	230	235	230
q15	523	474	470	470
q16	512	397	399	397
q17	1003	619	679	619
q18	8502	7923	7832	7832
q19	8475	1309	1411	1309
q20	673	323	311	311
q21	5172	3453	3300	3300
q22	397	348	348	348
Total cold run time: 121001 ms
Total hot run time: 41289 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4590	4430	4502	4430
q2	378	283	269	269
q3	3113	2878	2944	2878
q4	1988	1728	1769	1728
q5	5332	5475	5561	5475
q6	223	127	126	126
q7	2226	1844	1830	1830
q8	3240	3409	3383	3383
q9	8718	8738	8635	8635
q10	4129	3885	3874	3874
q11	601	511	506	506
q12	808	609	609	609
q13	15863	3044	3106	3044
q14	298	275	270	270
q15	520	498	479	479
q16	484	456	445	445
q17	1815	1497	1491	1491
q18	8036	8014	7354	7354
q19	1773	1573	1503	1503
q20	3038	1788	1786	1786
q21	7478	4772	4815	4772
q22	853	542	565	542
Total cold run time: 75504 ms
Total hot run time: 55429 ms

@feiniaofeiafei

feiniaofeiafei commented Jun 2, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@feiniaofeiafei

feiniaofeiafei commented Jun 3, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run external

@feiniaofeiafei

feiniaofeiafei commented Jun 3, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented Jun 3, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40610 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94ec6ee4434faa20675ca39ef64fcd740ff2e46d, data reload: false
------ Round 1 ----------------------------------
q1	17600	4356	4211	4211
q2	2017	195	193	193
q3	10436	1214	1078	1078
q4	10201	827	814	814
q5	7476	2710	2661	2661
q6	220	135	136	135
q7	960	607	602	602
q8	9221	2108	2122	2108
q9	9379	6695	6627	6627
q10	9493	3859	3901	3859
q11	486	239	245	239
q12	473	226	263	226
q13	17373	3288	3183	3183
q14	270	225	219	219
q15	523	481	471	471
q16	508	413	404	404
q17	995	726	760	726
q18	8258	7724	7789	7724
q19	6985	1371	1206	1206
q20	657	312	319	312
q21	5180	3272	4094	3272
q22	404	351	340	340
Total cold run time: 119115 ms
Total hot run time: 40610 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4582	4395	4405	4395
q2	370	269	282	269
q3	3100	2834	2886	2834
q4	1993	1703	1801	1703
q5	5355	5345	5485	5345
q6	220	127	123	123
q7	2190	1830	1791	1791
q8	3256	3367	3390	3367
q9	8601	8623	8597	8597
q10	3995	3857	3829	3829
q11	584	507	506	506
q12	817	658	644	644
q13	17064	3040	3124	3040
q14	317	279	271	271
q15	533	480	475	475
q16	480	439	430	430
q17	1785	1487	1496	1487
q18	8086	7966	7204	7204
q19	1797	1500	1650	1500
q20	3480	1762	1842	1762
q21	7931	4696	4636	4636
q22	666	532	525	525
Total cold run time: 77202 ms
Total hot run time: 54733 ms

@feiniaofeiafei

feiniaofeiafei commented Jun 24, 2024

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@doris-robot

doris-robot commented Jun 24, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6362644d11c65ad089ae799a19aecd01195d9d9, data reload: false
------ Round 1 ----------------------------------
q1	18234	4448	4388	4388
q2	2010	188	187	187
q3	10474	1150	1058	1058
q4	10184	800	820	800
q5	7490	2652	2606	2606
q6	219	132	131	131
q7	938	596	599	596
q8	9238	2030	2034	2030
q9	8764	6514	6408	6408
q10	9040	3731	3767	3731
q11	451	252	240	240
q12	449	234	227	227
q13	18826	2980	2986	2980
q14	264	226	223	223
q15	518	488	483	483
q16	518	378	380	378
q17	955	700	709	700
q18	8086	7573	7443	7443
q19	5239	1509	1478	1478
q20	639	320	335	320
q21	5024	3916	3906	3906
q22	399	346	349	346
Total cold run time: 117959 ms
Total hot run time: 40659 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4316	4204	4430	4204
q2	385	281	269	269
q3	2999	2772	2698	2698
q4	1917	1621	1656	1621
q5	5242	5288	5291	5288
q6	216	127	127	127
q7	2140	1773	1719	1719
q8	3187	3316	3301	3301
q9	8359	8317	8340	8317
q10	3895	3729	3668	3668
q11	570	488	500	488
q12	765	610	598	598
q13	17728	3000	3008	3000
q14	285	269	252	252
q15	518	481	470	470
q16	489	409	417	409
q17	1772	1462	1472	1462
q18	7703	7564	7323	7323
q19	1724	1473	1525	1473
q20	1962	1800	1804	1800
q21	4942	4806	4815	4806
q22	640	557	544	544
Total cold run time: 71754 ms
Total hot run time: 53837 ms

@doris-robot

doris-robot commented Jun 24, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 174053 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6362644d11c65ad089ae799a19aecd01195d9d9, data reload: false
query1	920	383	384	383
query2	6434	2418	2275	2275
query3	6638	202	198	198
query4	21728	17348	17444	17348
query5	4165	483	490	483
query6	253	169	176	169
query7	4602	307	304	304
query8	309	285	279	279
query9	8565	2484	2492	2484
query10	607	299	278	278
query11	10514	10040	10021	10021
query12	133	84	85	84
query13	1636	379	371	371
query14	9481	7613	7667	7613
query15	274	185	184	184
query16	8125	290	270	270
query17	1879	532	561	532
query18	2092	280	270	270
query19	192	159	153	153
query20	98	81	85	81
query21	206	135	130	130
query22	4288	4077	4088	4077
query23	33772	33078	33099	33078
query24	11909	2819	2799	2799
query25	654	389	360	360
query26	1755	155	157	155
query27	2961	323	319	319
query28	7751	2126	2112	2112
query29	1022	621	613	613
query30	268	151	151	151
query31	964	724	735	724
query32	94	55	61	55
query33	769	282	286	282
query34	942	484	511	484
query35	782	634	601	601
query36	1079	957	903	903
query37	198	75	72	72
query38	2894	2763	2736	2736
query39	844	808	802	802
query40	278	139	131	131
query41	56	55	50	50
query42	120	99	102	99
query43	589	548	556	548
query44	1227	737	747	737
query45	201	171	165	165
query46	1072	728	733	728
query47	1850	1808	1782	1782
query48	366	306	304	304
query49	1191	413	406	406
query50	761	386	396	386
query51	6907	6773	6855	6773
query52	109	92	94	92
query53	352	283	292	283
query54	976	464	453	453
query55	77	77	77	77
query56	296	275	287	275
query57	1157	1036	1072	1036
query58	287	266	259	259
query59	3545	3367	3095	3095
query60	290	300	290	290
query61	93	91	88	88
query62	657	448	461	448
query63	313	286	290	286
query64	9850	2280	1766	1766
query65	3162	3077	3139	3077
query66	1298	347	330	330
query67	15643	15199	14852	14852
query68	4624	563	558	558
query69	513	397	400	397
query70	1201	1073	1172	1073
query71	448	282	278	278
query72	6972	5532	5473	5473
query73	761	337	328	328
query74	5861	5619	5507	5507
query75	3460	2684	2641	2641
query76	2900	1015	922	922
query77	649	302	301	301
query78	10486	9750	9874	9750
query79	2124	521	517	517
query80	1143	507	555	507
query81	573	219	223	219
query82	773	107	104	104
query83	261	177	171	171
query84	238	87	89	87
query85	1586	287	271	271
query86	484	294	329	294
query87	3245	3077	3079	3077
query88	4121	2476	2444	2444
query89	469	381	380	380
query90	1798	195	196	195
query91	129	102	102	102
query92	68	52	52	52
query93	2138	522	537	522
query94	1236	203	199	199
query95	414	326	322	322
query96	595	284	281	281
query97	3245	3066	3009	3009
query98	210	202	202	202
query99	1212	844	821	821
Total cold run time: 278457 ms
Total hot run time: 174053 ms

@doris-robot

doris-robot commented Jun 24, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6362644d11c65ad089ae799a19aecd01195d9d9, data reload: false
query1	0.04	0.03	0.03
query2	0.10	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.08
query5	0.54	0.48	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.54	0.53
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.59	0.59	0.58
query14	0.78	0.79	0.81
query15	0.83	0.82	0.81
query16	0.36	0.37	0.36
query17	1.01	1.03	1.04
query18	0.24	0.23	0.25
query19	1.77	1.83	1.84
query20	0.01	0.00	0.00
query21	15.43	0.65	0.65
query22	4.39	6.55	2.33
query23	18.31	1.42	1.22
query24	2.10	0.24	0.22
query25	0.14	0.09	0.09
query26	0.27	0.18	0.18
query27	0.07	0.08	0.08
query28	13.22	1.03	1.01
query29	12.58	3.32	3.28
query30	0.26	0.06	0.05
query31	2.85	0.39	0.39
query32	3.35	0.47	0.48
query33	2.81	2.94	2.89
query34	17.10	4.42	4.46
query35	4.53	4.51	4.49
query36	0.65	0.46	0.46
query37	0.19	0.14	0.15
query38	0.16	0.14	0.16
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.59 s
Total hot run time: 31.15 s

morningman
morningman approved these changes Jun 25, 2024
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morrySnow
morrySnow merged commit 12c59f6 into apache:masterJun 25, 2024
@feiniaofeiafeifeiniaofeiafei mentioned this pull request Jun 25, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jun 26, 2024
@feiniaofeiafei@dataroaring
[Feat](create table) support generated column (#35284)
2039b07
This pr has added support for generated columns. Values of a generated
column are computed from an expression included in the column
definition. Here is an example:
mysql> CREATE TABLE products (
product_id INT,
price DECIMAL(10,2),
quantity INT,
total_value DECIMAL(10,2) GENERATED ALWAYS AS (price * quantity)
) DISTRIBUTED BY HASH(product_id) PROPERTIES ("replication_num" = "1");
mysql> insert into products values(1, 10.00, 10, default);
mysql> insert into products(product_id, price, quantity) values(1, 20.00, 10);
mysql> select * from products;
+------------+-------+----------+-------------+
| product_id | price | quantity | total_value |
+------------+-------+----------+-------------+
| 1 | 10.00 | 10 | 100.00 |
| 1 | 20.00 | 10 | 200.00 |
+------------+-------+----------+-------------+
related docs in apache/doris-website#715
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
morrySnow pushed a commit that referenced this pull request Jun 26, 2024
@feiniaofeiafei
[fix](generated column) static variables should not be used in Expre…
1c5926a
…ssionToExpr (#36824)
introduced by #35284
Fix generated column, static variables should not be used in
ExpressionToExpr. There is only one static variable slotRefMap in the
ExpressionToExpr class globally. It may be used by multiple threads at
the same time and assigned repeatedly, which is problematic. The same
reason applies to the modification of class slotRefRewriteRule.
No regression case added because this problem does not occur every time.
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@feiniaofeiafei@dataroaring
[fix](generated column) static variables should not be used in Expre…
940cbbe
…ssionToExpr (#36824)
introduced by #35284
Fix generated column, static variables should not be used in
ExpressionToExpr. There is only one static variable slotRefMap in the
ExpressionToExpr class globally. It may be used by multiple threads at
the same time and assigned repeatedly, which is problematic. The same
reason applies to the modification of class slotRefRewriteRule.
No regression case added because this problem does not occur every time.
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
luzhijing pushed a commit to apache/doris-website that referenced this pull request Jul 10, 2024
@feiniaofeiafei
[docs](ddl) add docs for generated column (#715)
f0fdc62
related code in apache/doris#35284
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
@feiniaofeiafeifeiniaofeiafei mentioned this pull request Mar 18, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Apr 14, 2025
@feiniaofeiafei
[fix](nereids) fix generate column need add cast in stream load (#49167)
0afc483
### What problem does this PR solve?
Related PR: #35284
Problem Summary:
Generate column need add cast(casting slot to its own type) in stream
load, because when loading data(stream load and other load), the slots
reading from files are string type. So we need to cast it to its own
type to avoid error.
github-actionsBot pushed a commit that referenced this pull request Apr 14, 2025
@feiniaofeiafeiYour Name
[fix](nereids) fix generate column need add cast in stream load (#49167)
30b64f4
### What problem does this PR solve?
Related PR: #35284
Problem Summary:
Generate column need add cast(casting slot to its own type) in stream
load, because when loading data(stream load and other load), the slots
reading from files are string type. So we need to cast it to its own
type to avoid error.
seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 17, 2025
@feiniaofeiafei@seawinde
[fix](nereids) fix generate column need add cast in stream load (apac…
83680e5
…he#49167)
### What problem does this PR solve?
Related PR: apache#35284
Problem Summary:
Generate column need add cast(casting slot to its own type) in stream
load, because when loading data(stream load and other load), the slots
reading from files are string type. So we need to cast it to its own
type to avoid error.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@feiniaofeiafeikoarz
[fix](nereids) fix generate column need add cast in stream load (apac…
5873088
…he#49167)
### What problem does this PR solve?
Related PR: apache#35284
Problem Summary:
Generate column need add cast(casting slot to its own type) in stream
load, because when loading data(stream load and other load), the slots
reading from files are string type. So we need to cast it to its own
type to avoid error.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@feiniaofeiafei@HappenLee
[Feat](create table) support generated column (apache#35284)
b9ccccb
This pr has added support for generated columns. Values of a generated
column are computed from an expression included in the column
definition. Here is an example:
mysql> CREATE TABLE products (
product_id INT,
price DECIMAL(10,2),
quantity INT,
total_value DECIMAL(10,2) GENERATED ALWAYS AS (price * quantity)
) DISTRIBUTED BY HASH(product_id) PROPERTIES ("replication_num" = "1");
mysql> insert into products values(1, 10.00, 10, default);
mysql> insert into products(product_id, price, quantity) values(1, 20.00, 10);
mysql> select * from products;
+------------+-------+----------+-------------+
| product_id | price | quantity | total_value |
+------------+-------+----------+-------------+
| 1 | 10.00 | 10 | 100.00 |
| 1 | 20.00 | 10 | 200.00 |
+------------+-------+----------+-------------+
related docs in apache/doris-website#715
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@feiniaofeiafei@HappenLee
[fix](generated column) static variables should not be used in Expre…
aa0b3c4
…ssionToExpr (apache#36824)
introduced by apache#35284
Fix generated column, static variables should not be used in
ExpressionToExpr. There is only one static variable slotRefMap in the
ExpressionToExpr class globally. It may be used by multiple threads at
the same time and assigned repeatedly, which is problematic. The same
reason applies to the modification of class slotRefRewriteRule.
No regression case added because this problem does not occur every time.
---------
Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@starocean999starocean999starocean999 approved these changes
@morrySnowmorrySnowmorrySnow approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.0-mergednot-merge/2.1reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@feiniaofeiafei@doris-robot@morrySnow@morningman@starocean999@dataroaring