[fix](job scheduler) specifies both startTime and immediate, it will trigger one fewer task execution by zddr · Pull Request #50624 · apache/doris

zddr · 2025-05-06T09:31:19Z

What problem does this PR solve?

CREATE MATERIALIZED VIEW mv1
REFRESH auto ON schedule every 1 day starts '2025-05-06 17:06:00'
DISTRIBUTED BY HASH(`k1`) BUCKETS 2
PROPERTIES (
'replication_num' = '1'
)
AS
select * from ta;

if current is 2025-05-06 17:00:00,

mv1 will has one task StartTime is 2025-05-06 17:00:00,next task StartTime is 2025-05-07 17:06:00

There should be another task with a start time of '2025-05-06 17:06:00'

Related PR: #xxx

Problem Summary:
fix MTMV specifies both startTime and immediate, it will trigger one fewer task execution

Release note

fix MTMV specifies both startTime and immediate, it will trigger one fewer task execution

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-05-06T09:31:25Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zddr · 2025-05-06T09:31:47Z

run buildall

doris-robot · 2025-05-06T10:14:52Z

TPC-H: Total hot run time: 33581 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa603d69c09008e7cd4cb91971ffb77b3de23371, data reload: false
------ Round 1 ----------------------------------
q1	25934	5013	4942	4942
q2	2056	272	185	185
q3	10409	1269	673	673
q4	10218	1005	501	501
q5	7517	2310	2337	2310
q6	174	163	132	132
q7	915	759	611	611
q8	9319	1290	1139	1139
q9	6968	5028	5046	5028
q10	6853	2309	1890	1890
q11	469	299	263	263
q12	342	342	209	209
q13	17760	3634	3077	3077
q14	223	218	201	201
q15	526	489	486	486
q16	406	416	363	363
q17	570	851	352	352
q18	7484	7047	7215	7047
q19	1605	945	542	542
q20	327	323	213	213
q21	3882	3292	2450	2450
q22	1045	1012	967	967
Total cold run time: 115002 ms
Total hot run time: 33581 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5168	5064	5107	5064
q2	226	327	229	229
q3	2279	2678	2356	2356
q4	1398	1812	1329	1329
q5	4468	4439	4462	4439
q6	218	167	127	127
q7	2011	1903	1749	1749
q8	2548	2585	2487	2487
q9	7194	7122	6835	6835
q10	2998	3177	2695	2695
q11	582	508	497	497
q12	657	736	590	590
q13	3452	3895	3217	3217
q14	273	299	257	257
q15	538	485	471	471
q16	422	491	441	441
q17	1116	1537	1364	1364
q18	7782	7523	7434	7434
q19	774	827	788	788
q20	1983	2013	1896	1896
q21	5121	4616	4526	4526
q22	1094	1082	1030	1030
Total cold run time: 52302 ms
Total hot run time: 49821 ms

doris-robot · 2025-05-06T10:27:05Z

TPC-DS: Total hot run time: 192511 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa603d69c09008e7cd4cb91971ffb77b3de23371, data reload: false
query1	1383	1101	1048	1048
query2	6311	1766	1748	1748
query3	10970	4580	4461	4461
query4	54154	25654	23532	23532
query5	5094	561	455	455
query6	361	216	217	216
query7	4880	514	287	287
query8	313	246	244	244
query9	5625	2561	2545	2545
query10	450	329	271	271
query11	15091	15135	15148	15135
query12	164	114	103	103
query13	1049	507	396	396
query14	10243	6390	6359	6359
query15	210	207	180	180
query16	7112	646	494	494
query17	1074	770	618	618
query18	1582	408	364	364
query19	201	188	164	164
query20	131	124	118	118
query21	199	127	107	107
query22	4428	4521	4390	4390
query23	34364	33499	33539	33499
query24	6569	2437	2446	2437
query25	456	479	404	404
query26	727	276	152	152
query27	2463	498	332	332
query28	3291	2073	2084	2073
query29	568	554	438	438
query30	274	218	191	191
query31	831	852	761	761
query32	70	60	67	60
query33	461	388	308	308
query34	767	861	531	531
query35	814	832	771	771
query36	960	979	927	927
query37	110	98	77	77
query38	4210	4262	4207	4207
query39	1471	1446	1439	1439
query40	218	117	107	107
query41	56	54	53	53
query42	121	109	117	109
query43	478	481	472	472
query44	1314	803	818	803
query45	179	185	168	168
query46	850	1049	659	659
query47	1904	1876	1806	1806
query48	376	416	301	301
query49	695	495	429	429
query50	646	707	405	405
query51	4136	4163	4131	4131
query52	106	112	107	107
query53	228	256	185	185
query54	582	568	506	506
query55	83	80	85	80
query56	316	301	282	282
query57	1166	1180	1118	1118
query58	273	263	252	252
query59	2724	2812	2598	2598
query60	323	328	319	319
query61	142	126	139	126
query62	735	705	735	705
query63	227	198	206	198
query64	1978	1038	701	701
query65	4307	4238	4225	4225
query66	736	401	299	299
query67	15809	15572	15312	15312
query68	7242	872	510	510
query69	525	305	263	263
query70	1167	1073	1092	1073
query71	490	320	293	293
query72	5511	4801	4925	4801
query73	1443	672	341	341
query74	8964	9102	8617	8617
query75	3983	3204	2682	2682
query76	4214	1176	735	735
query77	643	352	271	271
query78	10077	10193	9308	9308
query79	2147	795	570	570
query80	575	494	430	430
query81	485	250	224	224
query82	454	124	93	93
query83	239	247	229	229
query84	291	107	78	78
query85	876	357	301	301
query86	375	301	281	281
query87	4331	4433	4327	4327
query88	3477	2171	2160	2160
query89	392	331	276	276
query90	1816	199	210	199
query91	137	140	112	112
query92	71	59	58	58
query93	1689	939	561	561
query94	659	410	302	302
query95	362	294	286	286
query96	473	552	266	266
query97	3167	3187	3119	3119
query98	233	203	209	203
query99	1615	1425	1244	1244
Total cold run time: 298262 ms
Total hot run time: 192511 ms

doris-robot · 2025-05-06T10:32:16Z

ClickBench: Total hot run time: 28.73 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa603d69c09008e7cd4cb91971ffb77b3de23371, data reload: false
query1	0.03	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.18
query5	0.57	0.61	0.59
query6	1.18	0.71	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.50	0.52
query10	0.58	0.58	0.57
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.59	0.60
query14	0.81	0.79	0.80
query15	0.86	0.84	0.83
query16	0.38	0.38	0.39
query17	1.05	1.03	0.99
query18	0.20	0.19	0.19
query19	1.89	1.78	1.81
query20	0.01	0.01	0.01
query21	15.41	0.89	0.54
query22	0.78	1.24	0.70
query23	14.81	1.40	0.63
query24	7.56	1.50	0.53
query25	0.50	0.14	0.08
query26	0.68	0.16	0.12
query27	0.06	0.04	0.05
query28	9.55	0.85	0.41
query29	12.56	3.95	3.24
query30	0.25	0.10	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.45
query33	3.01	3.07	3.06
query34	15.80	5.10	4.48
query35	4.50	4.48	4.51
query36	0.68	0.50	0.50
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.76 s
Total hot run time: 28.73 s

zddr · 2025-05-12T06:51:08Z

run buildall

github-actions · 2025-05-12T06:52:24Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-12T06:52:26Z

PR approved by anyone and no changes requested.

zddr · 2025-05-12T07:09:24Z

run buildall

doris-robot · 2025-05-12T07:36:16Z

TPC-H: Total hot run time: 34314 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c81102dfeed9afa37052bae5c3db04184cde253, data reload: false
------ Round 1 ----------------------------------
q1	26438	5706	5040	5040
q2	2104	293	181	181
q3	10385	1222	719	719
q4	12732	995	547	547
q5	7539	2431	2322	2322
q6	184	163	134	134
q7	926	735	616	616
q8	9313	1275	1074	1074
q9	8270	5404	5284	5284
q10	6992	2376	1974	1974
q11	494	312	286	286
q12	365	364	225	225
q13	18259	3987	3103	3103
q14	278	227	213	213
q15	717	500	488	488
q16	874	438	402	402
q17	799	861	385	385
q18	8432	7285	7130	7130
q19	2837	1047	557	557
q20	335	342	225	225
q21	3802	3294	2462	2462
q22	1020	1033	947	947
Total cold run time: 123095 ms
Total hot run time: 34314 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5231	5060	5026	5026
q2	245	333	226	226
q3	2117	2811	2435	2435
q4	1399	1903	1436	1436
q5	4434	4299	4344	4299
q6	210	166	130	130
q7	2062	1876	1760	1760
q8	2703	2637	2573	2573
q9	6933	7089	7023	7023
q10	2989	3195	2780	2780
q11	577	519	509	509
q12	696	744	612	612
q13	3458	3935	3303	3303
q14	287	298	267	267
q15	525	481	471	471
q16	434	481	447	447
q17	1145	1564	1375	1375
q18	7671	7480	7235	7235
q19	850	793	815	793
q20	1969	2050	1923	1923
q21	5279	4790	4804	4790
q22	1109	1030	998	998
Total cold run time: 52323 ms
Total hot run time: 50411 ms

doris-robot · 2025-05-12T07:48:31Z

TPC-DS: Total hot run time: 193472 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c81102dfeed9afa37052bae5c3db04184cde253, data reload: false
query1	1400	1097	1081	1081
query2	6260	1794	1789	1789
query3	11023	4569	4448	4448
query4	54457	25754	23322	23322
query5	5103	528	459	459
query6	356	210	197	197
query7	4903	522	287	287
query8	313	257	244	244
query9	5718	2675	2677	2675
query10	435	314	259	259
query11	15096	15125	14742	14742
query12	163	119	106	106
query13	1049	532	412	412
query14	10124	6429	6376	6376
query15	211	205	184	184
query16	7127	671	492	492
query17	1094	737	592	592
query18	1528	421	315	315
query19	220	205	166	166
query20	118	125	122	122
query21	206	127	107	107
query22	4387	4419	4181	4181
query23	34221	33628	33583	33583
query24	6592	2435	2416	2416
query25	460	464	418	418
query26	674	278	152	152
query27	2251	518	352	352
query28	2920	2210	2169	2169
query29	573	568	430	430
query30	270	250	194	194
query31	860	874	780	780
query32	74	67	62	62
query33	460	364	313	313
query34	781	922	529	529
query35	779	842	733	733
query36	954	1019	913	913
query37	116	103	79	79
query38	4232	4260	4254	4254
query39	1524	1449	1433	1433
query40	209	122	110	110
query41	55	61	52	52
query42	133	119	110	110
query43	503	506	483	483
query44	1337	856	861	856
query45	179	178	194	178
query46	875	1036	651	651
query47	1815	1892	1833	1833
query48	391	438	330	330
query49	714	516	442	442
query50	677	711	418	418
query51	4332	4318	4185	4185
query52	111	114	107	107
query53	240	256	201	201
query54	601	584	507	507
query55	85	84	84	84
query56	302	327	318	318
query57	1200	1216	1103	1103
query58	268	279	256	256
query59	2644	2665	2617	2617
query60	339	340	334	334
query61	124	150	120	120
query62	713	732	684	684
query63	226	192	187	187
query64	1511	1023	676	676
query65	4401	4356	4262	4262
query66	709	403	312	312
query67	15839	15770	15572	15572
query68	7142	881	513	513
query69	543	314	314	314
query70	1208	1145	1101	1101
query71	511	323	299	299
query72	5711	4846	4982	4846
query73	1459	735	354	354
query74	9351	9052	9035	9035
query75	3854	3238	2722	2722
query76	4308	1182	733	733
query77	627	372	283	283
query78	10210	10010	9295	9295
query79	2256	806	568	568
query80	673	515	451	451
query81	491	261	219	219
query82	441	127	100	100
query83	251	242	233	233
query84	290	109	86	86
query85	779	360	306	306
query86	377	293	289	289
query87	4478	4482	4345	4345
query88	3325	2280	2274	2274
query89	402	315	284	284
query90	1819	199	210	199
query91	142	136	112	112
query92	79	59	57	57
query93	1961	945	571	571
query94	666	420	304	304
query95	363	296	276	276
query96	483	570	281	281
query97	3160	3225	3206	3206
query98	230	212	198	198
query99	1456	1411	1264	1264
Total cold run time: 298611 ms
Total hot run time: 193472 ms

doris-robot · 2025-05-12T07:53:43Z

ClickBench: Total hot run time: 29.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c81102dfeed9afa37052bae5c3db04184cde253, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.11	0.12
query3	0.25	0.20	0.19
query4	1.59	0.19	0.20
query5	0.58	0.60	0.59
query6	1.18	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.54	0.52
query10	0.57	0.58	0.57
query11	0.16	0.10	0.10
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	0.78	0.80	0.81
query15	0.87	0.85	0.84
query16	0.38	0.38	0.41
query17	1.02	1.04	0.99
query18	0.23	0.21	0.21
query19	1.93	1.80	1.81
query20	0.01	0.02	0.01
query21	15.40	0.89	0.53
query22	0.75	1.28	0.78
query23	14.71	1.37	0.64
query24	7.51	0.78	0.93
query25	0.55	0.15	0.05
query26	0.58	0.16	0.12
query27	0.06	0.05	0.05
query28	9.26	0.90	0.44
query29	12.54	4.10	3.29
query30	0.25	0.09	0.07
query31	2.82	0.61	0.38
query32	3.23	0.55	0.47
query33	2.99	3.09	3.06
query34	15.78	5.10	4.44
query35	4.49	4.50	4.46
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.15 s
Total hot run time: 29.19 s

zddr · 2025-05-13T07:15:35Z

run feut

github-actions · 2025-05-13T07:15:45Z

PR approved by at least one committer and no changes requested.

zddr · 2025-05-13T07:15:46Z

run external

…trigger one fewer task execution (#50624) ### What problem does this PR solve? CREATE MATERIALIZED VIEW mv1 REFRESH auto ON schedule every 1 day starts '2025-05-06 17:06:00' DISTRIBUTED BY HASH(`k1`) BUCKETS 2 PROPERTIES ( 'replication_num' = '1' ) AS select * from ta; if current is `2025-05-06 17:00:00`, mv1 will has one task StartTime is `2025-05-06 17:00:00`,next task StartTime is `2025-05-07 17:06:00` There should be another task with a start time of '2025-05-06 17:06:00' ### Release note fix MTMV specifies both startTime and immediate, it will trigger one fewer task execution

…te, it will trigger one fewer task execution #50624 (#50897) Cherry-picked from #50624 Co-authored-by: zhangdong <zhangdong@selectdb.com>

…te, it will trigger one fewer task execution #50624 (#50896) Cherry-picked from #50624 Co-authored-by: zhangdong <zhangdong@selectdb.com>

…trigger one fewer task execution (apache#50624) ### What problem does this PR solve? CREATE MATERIALIZED VIEW mv1 REFRESH auto ON schedule every 1 day starts '2025-05-06 17:06:00' DISTRIBUTED BY HASH(`k1`) BUCKETS 2 PROPERTIES ( 'replication_num' = '1' ) AS select * from ta; if current is `2025-05-06 17:00:00`, mv1 will has one task StartTime is `2025-05-06 17:00:00`,next task StartTime is `2025-05-07 17:06:00` There should be another task with a start time of '2025-05-06 17:06:00' ### Release note fix MTMV specifies both startTime and immediate, it will trigger one fewer task execution

zddr added 4 commits May 6, 2025 16:56

1
5d2c37d

1
ff0a784

1
0a94869

1
fa603d6

morrySnow reviewed May 6, 2025
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/job/base/JobExecutionConfiguration.java

morrySnow added usercase Important user case type label dev/2.1.x dev/3.0.x labels May 6, 2025

zddr marked this pull request as draft May 12, 2025 02:04

zddr added 2 commits May 12, 2025 12:17

1
504d6f8

1
e6b30f8

zddr marked this pull request as ready for review May 12, 2025 06:50

CalvinKirs previously approved these changes May 12, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 12, 2025

github-actionsBot added the reviewed label May 12, 2025

morrySnow reviewed May 12, 2025
View reviewed changes

Comment threadfe/fe-core/src/test/java/org/apache/doris/job/base/JobExecutionConfigurationTest.java Outdated

1
8c81102

zddr dismissed CalvinKirs’s stale review via 8c81102May 12, 2025 07:09

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label May 12, 2025

morrySnow approved these changes May 13, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 13, 2025

CalvinKirs approved these changes May 14, 2025
View reviewed changes

morrySnow merged commit d10dc4a into apache:masterMay 14, 2025

morrySnow changed the title ~~[fix](mtmv)fix MTMV specifies both startTime and immediate, it will trigger one fewer task execution~~[fix](job scheduler) specifies both startTime and immediate, it will trigger one fewer task execution May 14, 2025

github-actionsBot mentioned this pull request May 14, 2025
branch-3.0: [fix](job scheduler) specifies both startTime and immediate, it will trigger one fewer task execution #50624 #50896
Merged

github-actionsBot mentioned this pull request May 14, 2025
branch-2.1: [fix](job scheduler) specifies both startTime and immediate, it will trigger one fewer task execution #50624 #50897
Merged

yiguolei pushed a commit that referenced this pull request May 14, 2025
branch-2.1: [fix](job scheduler) specifies both startTime and immedia…
c4d0e1e
…te, it will trigger one fewer task execution #50624 (#50897) Cherry-picked from #50624 Co-authored-by: zhangdong <zhangdong@selectdb.com>

yiguolei added dev/2.1.10-merged and removed dev/2.1.x labels May 14, 2025

dataroaring pushed a commit that referenced this pull request May 16, 2025
branch-3.0: [fix](job scheduler) specifies both startTime and immedia…
d34aeb6
…te, it will trigger one fewer task execution #50624 (#50896) Cherry-picked from #50624 Co-authored-by: zhangdong <zhangdong@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 16, 2025

yiguolei added dev/2.1.11-merged and removed dev/2.1.10-merged labels May 17, 2025

gavinchou mentioned this pull request Jun 11, 2025
Release Note 3.0.6 #51585
Closed

Uh oh!

Conversation

zddr commented May 6, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented May 6, 2025

Uh oh!

zddr commented May 6, 2025

Uh oh!

Uh oh!

doris-robot commented May 6, 2025

Uh oh!

doris-robot commented May 6, 2025

Uh oh!

doris-robot commented May 6, 2025

Uh oh!

zddr commented May 12, 2025

Uh oh!

github-actionsBot commented May 12, 2025

Uh oh!

github-actionsBot commented May 12, 2025

Uh oh!

Uh oh!

zddr commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

zddr commented May 13, 2025

Uh oh!

github-actionsBot commented May 13, 2025

Uh oh!

zddr commented May 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zddr commented May 6, 2025•
edited
Loading