[feat](job)Implementing Job in Nereids by CalvinKirs · Pull Request #41391 · apache/doris

CalvinKirs · 2024-09-27T05:59:59Z

Proposed changes

The JOB's execution SQL is currently defined by an older CUP file, which causes some issues with lexical analysis in the new optimizer as it doesn't pass under the old optimizer. Since the JOB's underlying execution already uses the new optimizer, we're planning to fully migrate to ANTLR4 for consistency.

doris-robot · 2024-09-27T06:00:04Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman

LGTM

github-actions · 2024-09-29T04:35:05Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-29T04:35:06Z

PR approved by anyone and no changes requested.

CalvinKirs · 2024-10-06T04:00:30Z

run buildall

doris-robot · 2024-10-06T04:39:33Z

TPC-H: Total hot run time: 40888 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1742b2f82065d611f1911330bef9728e0d675304, data reload: false
------ Round 1 ----------------------------------
q1	17558	7388	7257	7257
q2	2021	282	269	269
q3	12133	1043	1134	1043
q4	10564	778	760	760
q5	7709	2847	2768	2768
q6	235	150	147	147
q7	1051	633	621	621
q8	9349	1897	1955	1897
q9	6569	6431	6383	6383
q10	6980	2311	2332	2311
q11	436	247	244	244
q12	411	226	227	226
q13	17789	2988	2990	2988
q14	239	210	208	208
q15	572	516	510	510
q16	643	590	583	583
q17	966	527	551	527
q18	7117	6727	6690	6690
q19	1350	1012	1069	1012
q20	468	206	203	203
q21	4049	3238	3364	3238
q22	1125	1003	1023	1003
Total cold run time: 109334 ms
Total hot run time: 40888 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7253	7249	7258	7249
q2	325	233	232	232
q3	3012	2914	2927	2914
q4	2082	1859	1792	1792
q5	5784	5766	5730	5730
q6	233	145	142	142
q7	2284	1829	1840	1829
q8	3313	3578	3437	3437
q9	8954	8820	8928	8820
q10	3638	3547	3520	3520
q11	576	497	487	487
q12	831	620	612	612
q13	11040	3245	3229	3229
q14	300	288	278	278
q15	579	529	530	529
q16	695	627	622	622
q17	1852	1654	1595	1595
q18	8258	7734	7694	7694
q19	1717	1522	1577	1522
q20	2118	1849	1905	1849
q21	5597	5456	5536	5456
q22	1174	1031	1055	1031
Total cold run time: 71615 ms
Total hot run time: 60569 ms

doris-robot · 2024-10-06T04:51:37Z

TPC-DS: Total hot run time: 192017 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1742b2f82065d611f1911330bef9728e0d675304, data reload: false
query1	860	395	402	395
query2	6254	2021	2059	2021
query3	8682	193	201	193
query4	33691	23464	23535	23464
query5	3593	471	469	469
query6	272	162	160	160
query7	4187	301	303	301
query8	289	226	214	214
query9	9302	2639	2663	2639
query10	466	281	276	276
query11	17795	15111	15243	15111
query12	153	99	98	98
query13	1615	472	444	444
query14	9521	7651	7504	7504
query15	254	182	182	182
query16	7952	460	467	460
query17	1624	618	602	602
query18	2079	317	325	317
query19	371	163	156	156
query20	119	112	113	112
query21	216	112	108	108
query22	4915	4660	4463	4463
query23	35030	34140	34065	34065
query24	11043	2852	2901	2852
query25	622	412	404	404
query26	1164	161	168	161
query27	2235	305	306	305
query28	7434	2396	2424	2396
query29	822	460	440	440
query30	254	153	152	152
query31	1024	828	786	786
query32	101	55	55	55
query33	776	307	307	307
query34	893	509	505	505
query35	859	739	724	724
query36	1086	951	948	948
query37	161	113	89	89
query38	4093	3995	3872	3872
query39	1498	1435	1406	1406
query40	206	96	98	96
query41	47	48	44	44
query42	120	93	97	93
query43	535	494	487	487
query44	1226	802	794	794
query45	196	167	173	167
query46	1150	706	717	706
query47	1914	1861	1815	1815
query48	422	353	363	353
query49	893	413	407	407
query50	827	428	435	428
query51	7008	6845	6921	6845
query52	103	92	88	88
query53	257	177	186	177
query54	1195	497	481	481
query55	79	76	79	76
query56	292	265	289	265
query57	1267	1151	1134	1134
query58	236	230	239	230
query59	3165	3045	2781	2781
query60	289	276	276	276
query61	105	100	104	100
query62	839	669	671	669
query63	220	181	184	181
query64	4009	646	600	600
query65	3268	3149	3198	3149
query66	844	303	316	303
query67	15840	15661	15584	15584
query68	4462	562	566	562
query69	516	301	305	301
query70	1213	1133	1139	1133
query71	348	276	289	276
query72	7383	4042	3978	3978
query73	795	358	358	358
query74	10133	8921	9011	8921
query75	3424	2691	2627	2627
query76	2988	948	839	839
query77	585	293	281	281
query78	10524	9664	9543	9543
query79	3150	603	604	603
query80	1655	452	462	452
query81	600	239	237	237
query82	698	134	137	134
query83	282	134	132	132
query84	277	83	80	80
query85	1384	295	287	287
query86	415	315	290	290
query87	4523	4410	4324	4324
query88	4020	2525	2484	2484
query89	418	293	299	293
query90	1959	187	190	187
query91	140	114	112	112
query92	62	47	48	47
query93	1980	572	558	558
query94	1029	303	297	297
query95	359	259	256	256
query96	617	283	290	283
query97	3386	3189	3195	3189
query98	224	202	197	197
query99	1563	1282	1271	1271
Total cold run time: 300969 ms
Total hot run time: 192017 ms

doris-robot · 2024-10-06T04:57:04Z

ClickBench: Total hot run time: 32.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1742b2f82065d611f1911330bef9728e0d675304, data reload: false
query1	0.05	0.05	0.04
query2	0.05	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.53	0.49	0.52
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.51	0.49
query10	0.53	0.55	0.57
query11	0.13	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.58	0.58
query14	2.70	2.75	2.75
query15	0.88	0.83	0.81
query16	0.38	0.37	0.38
query17	1.04	1.08	1.01
query18	0.19	0.20	0.20
query19	1.98	1.87	1.94
query20	0.02	0.01	0.01
query21	15.35	0.60	0.59
query22	2.31	2.62	2.05
query23	17.07	0.92	0.82
query24	3.07	1.00	1.38
query25	0.17	0.12	0.03
query26	0.62	0.14	0.14
query27	0.05	0.04	0.03
query28	10.42	1.11	1.08
query29	12.54	3.16	3.18
query30	0.24	0.06	0.06
query31	2.89	0.38	0.39
query32	3.27	0.45	0.46
query33	2.99	3.06	2.98
query34	16.82	4.44	4.44
query35	4.49	4.44	4.46
query36	0.66	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.04
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.3 s
Total hot run time: 32.76 s

CalvinKirs · 2024-10-06T05:12:27Z

run buildall

doris-robot · 2024-10-06T05:47:38Z

TPC-H: Total hot run time: 40872 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1742b2f82065d611f1911330bef9728e0d675304, data reload: false
------ Round 1 ----------------------------------
q1	17639	7401	7312	7312
q2	2014	286	269	269
q3	12212	1073	1192	1073
q4	10565	718	722	718
q5	7738	2872	2812	2812
q6	238	152	146	146
q7	1010	629	644	629
q8	9340	1907	1940	1907
q9	6516	6437	6393	6393
q10	6960	2254	2334	2254
q11	440	250	253	250
q12	409	216	220	216
q13	17785	2957	3037	2957
q14	236	211	207	207
q15	572	515	514	514
q16	652	576	595	576
q17	976	525	542	525
q18	7192	6680	6704	6680
q19	1361	944	1025	944
q20	498	207	198	198
q21	4082	3309	3350	3309
q22	1089	985	983	983
Total cold run time: 109524 ms
Total hot run time: 40872 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7239	7234	7209	7209
q2	330	231	233	231
q3	3000	2952	2949	2949
q4	2034	1866	1817	1817
q5	5763	5698	5714	5698
q6	233	151	139	139
q7	2275	1860	1822	1822
q8	3340	3565	3454	3454
q9	8918	8861	8890	8861
q10	3595	3529	3479	3479
q11	585	491	490	490
q12	821	654	654	654
q13	10720	3226	3226	3226
q14	305	277	290	277
q15	578	512	524	512
q16	702	665	655	655
q17	1851	1623	1576	1576
q18	8247	7755	7674	7674
q19	1699	1452	1476	1452
q20	2124	1832	1849	1832
q21	5591	5414	5320	5320
q22	1135	1012	1042	1012
Total cold run time: 71085 ms
Total hot run time: 60339 ms

doris-robot · 2024-10-06T05:59:43Z

TPC-DS: Total hot run time: 191587 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1742b2f82065d611f1911330bef9728e0d675304, data reload: false
query1	846	385	402	385
query2	6256	2019	2031	2019
query3	8690	195	200	195
query4	33695	23541	23548	23541
query5	3564	485	464	464
query6	290	168	174	168
query7	4198	306	305	305
query8	293	231	231	231
query9	9366	2649	2631	2631
query10	466	296	281	281
query11	17828	15119	15181	15119
query12	154	104	97	97
query13	1590	463	454	454
query14	9561	7587	7289	7289
query15	258	172	186	172
query16	8060	447	469	447
query17	1619	618	578	578
query18	2170	317	329	317
query19	371	168	164	164
query20	123	115	118	115
query21	222	110	106	106
query22	4766	4444	4595	4444
query23	35562	33873	33859	33859
query24	11010	2871	2891	2871
query25	629	406	399	399
query26	1351	163	173	163
query27	2511	296	301	296
query28	7350	2399	2406	2399
query29	886	438	450	438
query30	273	164	149	149
query31	1017	797	796	796
query32	90	53	54	53
query33	775	293	316	293
query34	898	503	518	503
query35	852	741	726	726
query36	1109	928	935	928
query37	156	85	85	85
query38	4057	3897	3910	3897
query39	1471	1393	1440	1393
query40	256	99	102	99
query41	47	46	44	44
query42	120	99	100	99
query43	519	494	482	482
query44	1179	808	796	796
query45	197	164	169	164
query46	1142	726	705	705
query47	1902	1855	1822	1822
query48	438	359	349	349
query49	909	420	410	410
query50	831	410	417	410
query51	7108	6814	7005	6814
query52	121	86	84	84
query53	251	178	179	178
query54	1254	465	483	465
query55	80	73	76	73
query56	285	264	258	258
query57	1272	1179	1166	1166
query58	243	231	245	231
query59	3194	2895	2898	2895
query60	305	276	272	272
query61	105	102	103	102
query62	833	647	653	647
query63	217	183	185	183
query64	4001	618	639	618
query65	3247	3196	3167	3167
query66	766	303	314	303
query67	15807	15430	15632	15430
query68	4474	561	549	549
query69	470	300	305	300
query70	1183	1159	1094	1094
query71	327	292	281	281
query72	7222	4216	4010	4010
query73	760	356	347	347
query74	10177	9030	9173	9030
query75	3453	2671	2673	2671
query76	2921	889	855	855
query77	590	298	305	298
query78	11108	9795	9603	9603
query79	2349	597	610	597
query80	1298	428	452	428
query81	596	239	239	239
query82	632	140	139	139
query83	265	133	131	131
query84	272	73	82	73
query85	1959	298	294	294
query86	487	301	288	288
query87	4526	4264	4311	4264
query88	4139	2450	2415	2415
query89	420	287	283	283
query90	1944	189	188	188
query91	149	108	115	108
query92	66	45	48	45
query93	2277	561	558	558
query94	910	300	303	300
query95	363	258	256	256
query96	618	288	280	280
query97	3276	3172	3140	3140
query98	222	202	193	193
query99	1542	1316	1294	1294
Total cold run time: 302007 ms
Total hot run time: 191587 ms

doris-robot · 2024-10-06T06:05:13Z

ClickBench: Total hot run time: 33.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1742b2f82065d611f1911330bef9728e0d675304, data reload: false
query1	0.04	0.05	0.04
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.53	0.52	0.54
query6	1.13	0.74	0.72
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.54	0.57	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.70	2.73	2.86
query15	0.90	0.83	0.83
query16	0.39	0.39	0.37
query17	1.04	0.99	1.01
query18	0.20	0.20	0.20
query19	1.90	1.88	2.02
query20	0.02	0.01	0.01
query21	15.36	0.60	0.59
query22	2.46	3.06	1.99
query23	16.96	0.83	0.80
query24	2.88	1.13	1.27
query25	0.23	0.11	0.12
query26	0.42	0.16	0.14
query27	0.04	0.03	0.05
query28	10.70	1.08	1.08
query29	12.50	3.21	3.18
query30	0.24	0.06	0.06
query31	2.87	0.37	0.38
query32	3.30	0.47	0.46
query33	3.03	3.00	3.02
query34	17.18	4.51	4.48
query35	4.54	4.52	4.45
query36	0.63	0.48	0.47
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.61 s
Total hot run time: 33.02 s

CalvinKirs · 2024-10-06T06:48:22Z

run buildall

doris-robot · 2024-10-06T07:17:48Z

TPC-H: Total hot run time: 40702 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f13bb930c7894e6561c1ccb289ddd3e23a33617, data reload: false
------ Round 1 ----------------------------------
q1	17558	7352	7245	7245
q2	2010	303	276	276
q3	12053	1057	1162	1057
q4	10542	670	736	670
q5	7741	2873	2785	2785
q6	239	152	148	148
q7	1034	627	607	607
q8	9366	1975	2007	1975
q9	6568	6396	6423	6396
q10	6985	2262	2268	2262
q11	449	249	243	243
q12	403	213	215	213
q13	17764	2983	2986	2983
q14	246	212	211	211
q15	577	530	524	524
q16	666	572	598	572
q17	964	616	555	555
q18	7091	6688	6710	6688
q19	1355	937	1047	937
q20	490	205	204	204
q21	3986	3281	3207	3207
q22	1115	944	972	944
Total cold run time: 109202 ms
Total hot run time: 40702 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7199	7238	7200	7200
q2	325	230	232	230
q3	2958	2922	2968	2922
q4	2050	1840	1811	1811
q5	5812	5728	5689	5689
q6	233	148	147	147
q7	2255	1871	1814	1814
q8	3344	3557	3435	3435
q9	8853	8877	8819	8819
q10	3575	3567	3525	3525
q11	579	491	486	486
q12	810	624	638	624
q13	9645	3176	3139	3139
q14	317	295	274	274
q15	568	520	516	516
q16	701	656	640	640
q17	1840	1636	1635	1635
q18	8194	7682	7583	7583
q19	1699	1558	1402	1402
q20	2108	1876	1880	1876
q21	5535	5469	5304	5304
q22	1097	1069	1013	1013
Total cold run time: 69697 ms
Total hot run time: 60084 ms

doris-robot · 2024-10-06T07:29:56Z

TPC-DS: Total hot run time: 192290 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f13bb930c7894e6561c1ccb289ddd3e23a33617, data reload: false
query1	926	416	399	399
query2	6306	2038	2028	2028
query3	8685	191	201	191
query4	33807	23549	23589	23549
query5	3549	478	457	457
query6	286	168	168	168
query7	4191	312	308	308
query8	286	227	236	227
query9	9200	2663	2662	2662
query10	474	284	281	281
query11	17654	15270	15158	15158
query12	147	101	97	97
query13	1585	461	442	442
query14	9864	7753	7114	7114
query15	264	167	175	167
query16	7698	447	462	447
query17	1698	623	599	599
query18	1767	328	326	326
query19	364	156	158	156
query20	118	114	115	114
query21	223	113	111	111
query22	4880	4674	4842	4674
query23	34904	33945	33911	33911
query24	10967	2803	2803	2803
query25	609	410	395	395
query26	1109	163	164	163
query27	2222	305	300	300
query28	7459	2424	2407	2407
query29	760	445	438	438
query30	284	167	161	161
query31	1029	835	786	786
query32	100	54	57	54
query33	766	289	304	289
query34	924	536	502	502
query35	857	735	726	726
query36	1112	963	962	962
query37	150	89	84	84
query38	4040	3910	3889	3889
query39	1519	1430	1455	1430
query40	210	95	98	95
query41	46	44	45	44
query42	120	96	94	94
query43	531	487	490	487
query44	1175	794	805	794
query45	195	165	163	163
query46	1145	716	737	716
query47	1988	1861	1897	1861
query48	433	367	345	345
query49	905	406	387	387
query50	819	417	412	412
query51	7058	6818	6865	6818
query52	99	84	84	84
query53	252	175	174	174
query54	1188	468	471	468
query55	74	73	75	73
query56	282	267	248	248
query57	1255	1158	1160	1158
query58	227	236	233	233
query59	3238	3012	3075	3012
query60	298	265	248	248
query61	107	100	103	100
query62	870	662	674	662
query63	221	186	184	184
query64	4184	636	635	635
query65	3310	3206	3189	3189
query66	943	299	303	299
query67	15897	16048	15882	15882
query68	4566	552	549	549
query69	456	300	290	290
query70	1190	1135	1040	1040
query71	339	272	277	272
query72	6940	3971	4031	3971
query73	767	349	347	347
query74	10068	9045	8978	8978
query75	3412	2713	2697	2697
query76	2675	912	883	883
query77	605	283	297	283
query78	10647	9635	9563	9563
query79	4852	579	602	579
query80	2582	446	443	443
query81	594	240	239	239
query82	724	142	137	137
query83	310	142	135	135
query84	285	83	80	80
query85	1670	293	281	281
query86	397	308	310	308
query87	4506	4269	4361	4269
query88	4332	2429	2409	2409
query89	402	289	293	289
query90	1891	184	181	181
query91	141	103	106	103
query92	65	47	46	46
query93	3119	558	545	545
query94	884	294	295	294
query95	352	255	257	255
query96	637	286	286	286
query97	3249	3192	3178	3178
query98	228	224	204	204
query99	1513	1316	1320	1316
Total cold run time: 304251 ms
Total hot run time: 192290 ms

doris-robot · 2024-10-06T07:35:25Z

ClickBench: Total hot run time: 33.36 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f13bb930c7894e6561c1ccb289ddd3e23a33617, data reload: false
query1	0.05	0.05	0.04
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.09	0.09
query5	0.54	0.51	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.49	0.49
query10	0.57	0.55	0.55
query11	0.14	0.12	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.72	2.82	2.86
query15	0.91	0.82	0.83
query16	0.41	0.37	0.41
query17	1.06	1.01	1.00
query18	0.21	0.20	0.20
query19	1.93	1.86	2.00
query20	0.01	0.01	0.01
query21	15.35	0.59	0.60
query22	2.74	2.53	2.44
query23	17.02	1.17	0.82
query24	2.84	0.98	1.52
query25	0.18	0.09	0.13
query26	0.60	0.13	0.14
query27	0.04	0.05	0.05
query28	10.66	1.08	1.07
query29	12.60	3.19	3.16
query30	0.24	0.07	0.06
query31	2.88	0.38	0.39
query32	3.26	0.47	0.47
query33	3.03	2.97	3.05
query34	17.05	4.45	4.57
query35	4.53	4.53	4.56
query36	0.66	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 107.11 s
Total hot run time: 33.36 s

CalvinKirs · 2024-10-06T09:45:11Z

run buildall

doris-robot · 2024-10-06T10:23:57Z

TPC-H: Total hot run time: 40382 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be25a1f8b8ac57f23664fed845a675c19b1825bc, data reload: false
------ Round 1 ----------------------------------
q1	17574	7345	7231	7231
q2	2012	278	267	267
q3	12241	1042	1166	1042
q4	10568	722	717	717
q5	7742	2848	2851	2848
q6	234	146	144	144
q7	1013	644	632	632
q8	9354	1881	1927	1881
q9	6502	6414	6350	6350
q10	7003	2281	2288	2281
q11	442	251	259	251
q12	407	222	218	218
q13	17774	2968	3005	2968
q14	234	203	215	203
q15	564	540	538	538
q16	657	604	584	584
q17	962	554	485	485
q18	7191	6599	6763	6599
q19	1347	950	1034	950
q20	492	204	202	202
q21	3978	3121	3035	3035
q22	1101	973	956	956
Total cold run time: 109392 ms
Total hot run time: 40382 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7262	7187	7225	7187
q2	327	228	233	228
q3	2936	2971	2936	2936
q4	2031	1807	1819	1807
q5	5786	5752	5747	5747
q6	232	146	141	141
q7	2227	1878	1799	1799
q8	3373	3580	3442	3442
q9	8937	8892	8858	8858
q10	3603	3529	3540	3529
q11	584	491	489	489
q12	801	650	670	650
q13	10418	3219	3173	3173
q14	305	286	285	285
q15	575	528	512	512
q16	705	635	662	635
q17	1844	1634	1611	1611
q18	8337	7891	7552	7552
q19	1710	1407	1370	1370
q20	2106	1877	1881	1877
q21	5605	5283	5315	5283
q22	1120	1036	1049	1036
Total cold run time: 70824 ms
Total hot run time: 60147 ms

doris-robot · 2024-10-06T10:36:01Z

TPC-DS: Total hot run time: 192027 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be25a1f8b8ac57f23664fed845a675c19b1825bc, data reload: false
query1	915	396	384	384
query2	6237	2052	2015	2015
query3	8679	195	202	195
query4	33581	23377	23391	23377
query5	3406	470	471	470
query6	271	169	163	163
query7	4190	303	315	303
query8	285	214	215	214
query9	9597	2636	2638	2636
query10	471	299	293	293
query11	17939	15067	15150	15067
query12	155	96	96	96
query13	1587	472	444	444
query14	9524	7489	7452	7452
query15	245	171	175	171
query16	8012	526	478	478
query17	1677	618	593	593
query18	2025	339	316	316
query19	370	159	154	154
query20	125	113	112	112
query21	222	109	108	108
query22	4795	4684	4396	4396
query23	34926	34211	34100	34100
query24	10976	2828	2869	2828
query25	615	416	416	416
query26	1148	161	162	161
query27	2281	318	301	301
query28	7474	2410	2386	2386
query29	820	455	427	427
query30	268	149	157	149
query31	1064	799	816	799
query32	97	56	55	55
query33	757	303	303	303
query34	930	526	524	524
query35	880	717	726	717
query36	1117	938	952	938
query37	160	92	93	92
query38	4130	4045	3880	3880
query39	1482	1431	1416	1416
query40	209	100	100	100
query41	49	49	44	44
query42	113	100	100	100
query43	537	497	476	476
query44	1214	790	806	790
query45	195	162	164	162
query46	1194	731	716	716
query47	1956	1855	1810	1810
query48	452	365	348	348
query49	901	425	407	407
query50	830	423	429	423
query51	7101	7057	6906	6906
query52	105	94	87	87
query53	266	186	182	182
query54	1213	477	506	477
query55	78	80	77	77
query56	278	286	295	286
query57	1265	1187	1162	1162
query58	232	225	237	225
query59	3075	2868	2849	2849
query60	294	282	283	282
query61	109	102	105	102
query62	900	673	667	667
query63	222	191	186	186
query64	4043	640	622	622
query65	3232	3153	3172	3153
query66	828	302	307	302
query67	15757	15703	15548	15548
query68	4029	581	554	554
query69	518	304	306	304
query70	1172	1156	1130	1130
query71	383	270	279	270
query72	7293	4042	3987	3987
query73	797	353	350	350
query74	10257	8924	9091	8924
query75	3454	2745	2643	2643
query76	2971	907	923	907
query77	637	320	311	311
query78	10626	9675	9555	9555
query79	1747	604	590	590
query80	2696	475	474	474
query81	605	234	240	234
query82	702	138	141	138
query83	306	136	138	136
query84	288	82	78	78
query85	1668	309	296	296
query86	436	286	309	286
query87	4605	4389	4354	4354
query88	3276	2452	2407	2407
query89	405	283	294	283
query90	2140	191	186	186
query91	142	110	108	108
query92	65	49	49	49
query93	1327	546	546	546
query94	1230	292	294	292
query95	364	256	266	256
query96	656	286	284	284
query97	3320	3239	3170	3170
query98	212	202	205	202
query99	1605	1305	1311	1305
Total cold run time: 300013 ms
Total hot run time: 192027 ms

morrySnow · 2024-10-10T03:42:19Z

 | SHOW CREATE MATERIALIZED VIEW mvName=multipartIdentifier #showCreateMTMV
 ;
-
+jobScheduleStatement


Suggested change
jobScheduleStatement
supportedJobStatement

morrySnow · 2024-10-10T03:43:09Z

 long base = TimeUnit.MINUTES.toMillis(connectionAgeBase);
 long now = System.currentTimeMillis();
- long rand = random.nextLong(base);
+ long rand = random.nextLong() % base;


already fixed, please rebase to newest master

morrySnow · 2024-10-10T03:51:40Z

+ Optional<Long> interval = ctx.timeInterval == null ? Optional.empty() :
+ Optional.of(Long.valueOf(ctx.timeInterval.getText()));
+ Optional<String> intervalUnit = ctx.timeUnit == null ? Optional.empty() : Optional.of(ctx.timeUnit.getText());
+ String commentSpec = ctx.commentSpec() == null ? "''" : ctx.commentSpec().STRING_LITERAL().getText();


could u have add override visitCommentSpec, let it return processed comment string?

morrySnow · 2024-10-10T04:02:30Z

+ sql """
+ SET enable_fallback_to_original_planner=false;
+ """


this variable has been removed and u do not need to add it

morrySnow · 2024-10-10T04:04:11Z

 ;
-
+jobScheduleStatement
+ : CREATE JOB label=multipartIdentifier ON SCHEDULE


remove create job syntax in sql_parser.cup and remove CreateJobStmt.java

I want to directly pick to 2.1, as 2.1 allows for a rollback. Therefore, this change will be provided in the next PR.:)

morningman

LGTM

morningman · 2024-10-11T04:54:13Z

run buildall

github-actions · 2024-10-11T04:54:27Z

PR approved by at least one committer and no changes requested.

CalvinKirs · 2024-10-11T05:55:16Z

run feut

morningman

LGTM

CalvinKirs · 2024-10-11T06:14:06Z

run cloud_p0

## Proposed changes The JOB's execution SQL is currently defined by an older CUP file, which causes some issues with lexical analysis in the new optimizer as it doesn't pass under the old optimizer. Since the JOB's underlying execution already uses the new optimizer, we're planning to fully migrate to ANTLR4 for consistency.

## Proposed changes The JOB's execution SQL is currently defined by an older CUP file, which causes some issues with lexical analysis in the new optimizer as it doesn't pass under the old optimizer. Since the JOB's underlying execution already uses the new optimizer, we're planning to fully migrate to ANTLR4 for consistency. (cherry picked from commit 334b473)

## Proposed changes The JOB's execution SQL is currently defined by an older CUP file, which causes some issues with lexical analysis in the new optimizer as it doesn't pass under the old optimizer. Since the JOB's underlying execution already uses the new optimizer, we're planning to fully migrate to ANTLR4 for consistency. (cherry picked from commit 334b473) #41391

bp #41391

## Proposed changes The JOB's execution SQL is currently defined by an older CUP file, which causes some issues with lexical analysis in the new optimizer as it doesn't pass under the old optimizer. Since the JOB's underlying execution already uses the new optimizer, we're planning to fully migrate to ANTLR4 for consistency.

morningman added dev/2.1.x dev/3.0.x labels Sep 27, 2024

morningman reviewed Sep 27, 2024
View reviewed changes

morningman previously approved these changes Sep 29, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 29, 2024

github-actionsBot added the reviewed label Sep 29, 2024

CalvinKirs requested a review from morrySnow September 29, 2024 05:06

CalvinKirs dismissed morningman’s stale review via 65b794aSeptember 30, 2024 07:58

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 30, 2024

[feat](job)Implementing Job using antlr4
2f13bb9

CalvinKirs force-pushed the master-new-job branch from 8ddd998 to 2f13bb9Compare October 6, 2024 06:48

fix test
be25a1f

morrySnow reviewed Oct 10, 2024
View reviewed changes

CalvinKirsand others added 2 commits October 10, 2024 14:04

Merge branch 'master' into master-new-job
20e6f03

minor change
ef45c6b

CalvinKirs dismissed 924060929’s stale review via ef45c6bOctober 10, 2024 06:33

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 10, 2024

minor change
0e68131

morrySnow changed the title ~~[feat](job)Implementing Job using antlr4~~[feat](job)Implementing Job in NereidsOct 10, 2024

morningman approved these changes Oct 11, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024

morningman approved these changes Oct 11, 2024
View reviewed changes

starocean999 approved these changes Oct 11, 2024
View reviewed changes

starocean999 merged commit 334b473 into apache:masterOct 11, 2024

CalvinKirs deleted the master-new-job branch October 12, 2024 02:06

CalvinKirs mentioned this pull request Oct 15, 2024
[branch-2.1][feat](job)Implementing Job in Nereids (#41391) #41859
Closed

CalvinKirs mentioned this pull request Oct 17, 2024
[branch-2.1][feat](job)Implementing Job in Nereids (#41391) #42012
Merged

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Oct 17, 2024

morningman mentioned this pull request Oct 18, 2024
[feat](job)Implementing Job in Nereids (#41391) #41861
Merged

morningman pushed a commit that referenced this pull request Oct 18, 2024
[feat](job)Implementing Job in Nereids (#41391) (#41861)
1a61cb4
bp #41391

morningman added dev/3.0.3-merged and removed dev/3.0.x labels Oct 18, 2024

Uh oh!

Conversation

CalvinKirs commented Sep 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Sep 27, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 29, 2024

Uh oh!

github-actionsBot commented Sep 29, 2024

Uh oh!

CalvinKirs commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

CalvinKirs commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

CalvinKirs commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

CalvinKirs commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

doris-robot commented Oct 6, 2024

Uh oh!

morrySnowOct 10, 2024

Choose a reason for hiding this comment

Uh oh!

morrySnowOct 10, 2024

Choose a reason for hiding this comment

Uh oh!

morrySnowOct 10, 2024

Choose a reason for hiding this comment

Uh oh!

morrySnowOct 10, 2024

Choose a reason for hiding this comment

Uh oh!

morrySnowOct 10, 2024

Choose a reason for hiding this comment

Uh oh!

CalvinKirsOct 10, 2024

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Oct 11, 2024

Uh oh!

github-actionsBot commented Oct 11, 2024

Uh oh!

CalvinKirs commented Oct 11, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

CalvinKirs commented Sep 27, 2024•
edited
Loading