Skip to content

[enhance](mtmv)cache table snapshot in refresh context - #50855

Merged
morrySnow merged 6 commits into
apache:masterfrom
zddr:snapshot_context
May 20, 2025
Merged

[enhance](mtmv)cache table snapshot in refresh context#50855
morrySnow merged 6 commits into
apache:masterfrom
zddr:snapshot_context

Conversation

@zddr

@zddrzddr commented May 13, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Assuming the SQL definition below, there are two tables (t1, t2) and one materialized view (mv1).

When determining whether mv1 is synchronized with its base tables, the system compares:

Whether mvp1 is synchronized with partition p1 and base table t2

Whether mvp2 is synchronized with partition p2 and base table t2

Optimization in this PR:

The original logic would fetch t2's snapshot information twice. This PR improves efficiency by caching t2's snapshot in the refresh context to avoid redundant retrieval.

CREATE TABLE t1
(
k2 TINYINT,
k3 INT not null
)
PARTITION BY LIST(`k3`)
(
PARTITION `p1` VALUES IN ('1'),
PARTITION `p2` VALUES IN ('2')
)
CREATE TABLE t2
(
k2 TINYINT,
k3 INT not null
);
create materialized view mv1 partition by(k3)
as
select * from t1 join t2;
mv1 will has two partition mvp1,mvp2

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented May 13, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr

zddr commented May 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

morrySnow
morrySnow reviewed May 13, 2025
View reviewed changes

@morrySnowmorrySnow left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add ut

@zddr

zddr commented May 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@zddr
zddr requested a review from morrySnowMay 14, 2025 02:40
@zddr

zddr commented May 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33837 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91e6799a6f9d06daddf886e174047fd46b617a13, data reload: false
------ Round 1 ----------------------------------
q1	26323	5090	5030	5030
q2	2077	278	192	192
q3	10386	1270	689	689
q4	10214	1009	512	512
q5	7535	2390	2327	2327
q6	184	166	135	135
q7	913	766	622	622
q8	9336	1319	1134	1134
q9	6770	5080	5050	5050
q10	6869	2350	1901	1901
q11	513	278	266	266
q12	348	354	217	217
q13	17788	3636	3092	3092
q14	232	229	210	210
q15	545	468	494	468
q16	449	440	381	381
q17	599	866	345	345
q18	7541	7253	7129	7129
q19	1681	967	566	566
q20	325	343	216	216
q21	4202	3283	2407	2407
q22	1023	1018	948	948
Total cold run time: 115853 ms
Total hot run time: 33837 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5204	5126	5112	5112
q2	245	331	230	230
q3	2183	2718	2317	2317
q4	1416	1780	1399	1399
q5	4444	4447	4438	4438
q6	212	175	128	128
q7	1988	1974	1763	1763
q8	2589	2640	2528	2528
q9	7167	7218	6829	6829
q10	3069	3184	2735	2735
q11	592	522	510	510
q12	697	767	619	619
q13	3581	3906	3281	3281
q14	287	308	297	297
q15	530	485	480	480
q16	448	475	429	429
q17	1169	1601	1390	1390
q18	7791	7596	7478	7478
q19	833	813	856	813
q20	2032	1969	1903	1903
q21	5103	4735	4740	4735
q22	1104	1068	979	979
Total cold run time: 52684 ms
Total hot run time: 50393 ms

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193941 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91e6799a6f9d06daddf886e174047fd46b617a13, data reload: false
query1	1428	1100	1061	1061
query2	6370	1887	1887	1887
query3	11016	4477	4509	4477
query4	52454	25717	22943	22943
query5	5188	551	451	451
query6	350	207	201	201
query7	4966	546	296	296
query8	318	261	242	242
query9	5851	2664	2656	2656
query10	432	339	273	273
query11	15316	14947	14859	14859
query12	160	114	113	113
query13	1078	535	401	401
query14	10161	6391	6436	6391
query15	218	200	188	188
query16	7072	705	517	517
query17	1101	773	603	603
query18	1567	436	339	339
query19	213	203	175	175
query20	140	122	116	116
query21	218	134	108	108
query22	4599	4635	4366	4366
query23	34515	33655	33641	33641
query24	6544	2477	2476	2476
query25	485	476	419	419
query26	714	290	155	155
query27	2257	524	335	335
query28	3040	2182	2147	2147
query29	590	577	465	465
query30	278	239	189	189
query31	850	862	777	777
query32	71	66	60	60
query33	472	383	307	307
query34	817	902	541	541
query35	819	839	759	759
query36	936	998	896	896
query37	113	108	78	78
query38	4234	4160	4298	4160
query39	1504	1452	1466	1452
query40	208	125	110	110
query41	54	57	53	53
query42	128	116	116	116
query43	521	519	500	500
query44	1415	836	840	836
query45	187	175	166	166
query46	895	1085	664	664
query47	1828	1881	1835	1835
query48	404	463	333	333
query49	725	551	421	421
query50	710	745	409	409
query51	4253	4319	4273	4273
query52	118	113	117	113
query53	245	269	201	201
query54	615	604	564	564
query55	86	87	87	87
query56	299	312	319	312
query57	1191	1182	1142	1142
query58	265	261	277	261
query59	2741	2805	2697	2697
query60	325	317	331	317
query61	157	130	132	130
query62	744	766	664	664
query63	268	190	185	185
query64	1734	1063	712	712
query65	4312	4213	4214	4213
query66	717	407	310	310
query67	15769	15844	15505	15505
query68	6880	919	522	522
query69	538	303	283	283
query70	1229	1121	1052	1052
query71	506	330	287	287
query72	5816	4832	5021	4832
query73	1461	706	359	359
query74	9198	8978	8988	8978
query75	4010	3310	2724	2724
query76	4299	1240	771	771
query77	649	391	281	281
query78	10108	10151	9402	9402
query79	2482	847	591	591
query80	610	619	443	443
query81	495	257	223	223
query82	437	131	98	98
query83	320	259	246	246
query84	295	104	88	88
query85	806	362	319	319
query86	369	303	302	302
query87	4365	4489	4407	4407
query88	3411	2303	2312	2303
query89	407	327	285	285
query90	1812	211	210	210
query91	146	148	114	114
query92	79	64	57	57
query93	1908	969	578	578
query94	670	450	302	302
query95	365	296	292	292
query96	495	627	287	287
query97	3208	3207	3160	3160
query98	235	217	206	206
query99	1434	1403	1290	1290
Total cold run time: 298412 ms
Total hot run time: 193941 ms

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91e6799a6f9d06daddf886e174047fd46b617a13, data reload: false
query1	0.04	0.03	0.03
query2	0.12	0.11	0.12
query3	0.26	0.20	0.20
query4	1.60	0.20	0.19
query5	0.60	0.57	0.57
query6	1.20	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.57	0.51	0.53
query10	0.56	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.12	0.13
query13	0.61	0.60	0.59
query14	0.79	0.80	0.80
query15	0.88	0.84	0.86
query16	0.39	0.39	0.39
query17	1.08	1.06	1.05
query18	0.22	0.21	0.21
query19	1.93	1.81	1.82
query20	0.01	0.01	0.01
query21	15.40	0.90	0.54
query22	0.74	1.21	0.95
query23	14.68	1.33	0.62
query24	6.92	1.94	0.63
query25	0.54	0.22	0.22
query26	0.68	0.16	0.14
query27	0.05	0.05	0.04
query28	9.60	0.85	0.44
query29	12.55	4.09	3.35
query30	0.25	0.09	0.07
query31	2.83	0.59	0.40
query32	3.22	0.56	0.46
query33	3.08	3.04	3.25
query34	15.74	5.11	4.54
query35	4.53	4.50	4.49
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.07	0.04	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 103.16 s
Total hot run time: 29.6 s

@zddr

zddr commented May 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33504 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 41a58fc55ada2474be9b75c45bfa3c1d1614ad16, data reload: false
------ Round 1 ----------------------------------
q1	25958	5064	4939	4939
q2	2068	269	178	178
q3	10659	1250	685	685
q4	10254	1378	552	552
q5	7584	2407	2318	2318
q6	181	167	136	136
q7	913	765	639	639
q8	9333	1181	1059	1059
q9	7165	5060	5102	5060
q10	6877	2298	1873	1873
q11	476	282	269	269
q12	338	343	207	207
q13	17780	3671	3068	3068
q14	231	227	215	215
q15	523	503	492	492
q16	422	438	369	369
q17	586	844	349	349
q18	7486	7152	6995	6995
q19	2010	981	537	537
q20	333	322	217	217
q21	3787	3352	2381	2381
q22	1035	994	966	966
Total cold run time: 115999 ms
Total hot run time: 33504 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5207	5061	5052	5052
q2	233	326	230	230
q3	2191	2665	2280	2280
q4	1331	1768	1333	1333
q5	4431	4402	4390	4390
q6	208	172	125	125
q7	2002	1917	1739	1739
q8	2627	2549	2491	2491
q9	7142	7213	6974	6974
q10	2949	3278	2785	2785
q11	575	526	489	489
q12	665	752	652	652
q13	3552	3931	3360	3360
q14	288	315	266	266
q15	524	467	499	467
q16	449	505	456	456
q17	1118	1519	1398	1398
q18	7710	7482	7470	7470
q19	774	802	965	802
q20	2015	2041	1913	1913
q21	5057	4645	4787	4645
q22	1066	1029	1052	1029
Total cold run time: 52114 ms
Total hot run time: 50346 ms

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193779 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41a58fc55ada2474be9b75c45bfa3c1d1614ad16, data reload: false
query1	1418	1083	1043	1043
query2	6227	1791	1811	1791
query3	11045	4349	4510	4349
query4	53524	25364	23521	23521
query5	5285	473	462	462
query6	385	210	193	193
query7	5140	490	294	294
query8	346	265	236	236
query9	6791	2649	2642	2642
query10	445	342	269	269
query11	15822	14986	14897	14897
query12	168	110	106	106
query13	1150	528	440	440
query14	10107	6212	6234	6212
query15	204	192	177	177
query16	7179	639	503	503
query17	1059	750	580	580
query18	1504	395	311	311
query19	197	204	156	156
query20	124	125	117	117
query21	210	160	104	104
query22	4675	4578	4492	4492
query23	34118	33687	33534	33534
query24	6614	2479	2436	2436
query25	468	506	440	440
query26	665	274	154	154
query27	2415	520	350	350
query28	3281	2173	2174	2173
query29	601	590	491	491
query30	276	220	192	192
query31	850	853	765	765
query32	79	65	70	65
query33	454	382	329	329
query34	771	873	547	547
query35	793	831	768	768
query36	976	1020	921	921
query37	114	102	118	102
query38	4138	4279	4165	4165
query39	1511	1485	1486	1485
query40	209	118	110	110
query41	55	55	52	52
query42	135	150	112	112
query43	513	526	465	465
query44	1335	834	843	834
query45	178	174	168	168
query46	857	1034	638	638
query47	1871	1887	1824	1824
query48	406	431	330	330
query49	717	517	435	435
query50	650	704	411	411
query51	4199	4199	4215	4199
query52	110	105	99	99
query53	222	258	190	190
query54	602	575	518	518
query55	91	81	84	81
query56	330	310	305	305
query57	1182	1217	1131	1131
query58	274	269	263	263
query59	2678	2763	2664	2664
query60	350	353	300	300
query61	126	130	125	125
query62	738	781	702	702
query63	239	194	193	193
query64	1436	1008	694	694
query65	4323	4237	4236	4236
query66	726	393	307	307
query67	15851	15654	15722	15654
query68	7571	879	531	531
query69	562	307	267	267
query70	1154	1113	1103	1103
query71	497	328	311	311
query72	5988	4808	4773	4773
query73	1176	649	355	355
query74	9460	8795	9068	8795
query75	3916	3266	2830	2830
query76	4242	1202	747	747
query77	631	373	295	295
query78	9942	10250	9340	9340
query79	2034	787	581	581
query80	634	544	449	449
query81	471	254	223	223
query82	441	123	99	99
query83	413	249	231	231
query84	288	106	88	88
query85	848	357	312	312
query86	412	303	273	273
query87	4413	4479	4326	4326
query88	3497	2258	2265	2258
query89	409	314	286	286
query90	1974	206	208	206
query91	144	145	114	114
query92	79	60	57	57
query93	1357	925	580	580
query94	665	384	300	300
query95	376	286	279	279
query96	492	573	286	286
query97	3165	3255	3143	3143
query98	234	203	202	202
query99	1401	1393	1287	1287
Total cold run time: 300418 ms
Total hot run time: 193779 ms

@doris-robot

doris-robot commented May 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41a58fc55ada2474be9b75c45bfa3c1d1614ad16, data reload: false
query1	0.04	0.03	0.04
query2	0.12	0.11	0.12
query3	0.25	0.20	0.19
query4	1.58	0.20	0.19
query5	0.60	0.59	0.60
query6	1.22	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.52	0.52
query10	0.57	0.59	0.57
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.59
query14	0.79	0.81	0.81
query15	0.88	0.85	0.86
query16	0.37	0.38	0.38
query17	1.04	1.06	1.05
query18	0.23	0.21	0.21
query19	1.88	1.79	1.83
query20	0.01	0.01	0.02
query21	15.39	0.92	0.55
query22	0.75	1.05	0.61
query23	15.12	1.38	0.61
query24	7.60	0.97	0.44
query25	0.48	0.25	0.20
query26	0.62	0.16	0.13
query27	0.05	0.05	0.05
query28	8.65	0.90	0.44
query29	12.55	3.98	3.33
query30	0.25	0.09	0.07
query31	2.82	0.58	0.39
query32	3.23	0.55	0.47
query33	3.03	3.09	3.19
query34	15.69	5.04	4.48
query35	4.53	4.53	4.51
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.04 s
Total hot run time: 29.05 s

morrySnow
morrySnow reviewed May 16, 2025
View reviewed changes
Comment threadfe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVPartitionUtilTest.java
seawinde
seawinde reviewed May 16, 2025
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java
@github-actions

github-actionsBot commented May 16, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@zddr

zddr commented May 16, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented May 16, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34310 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46f8d99fd98eeffb86d3fd20ab0ac2d0d4a090fc, data reload: false
------ Round 1 ----------------------------------
q1	26584	5075	5305	5075
q2	2104	279	178	178
q3	10407	1259	676	676
q4	11177	999	526	526
q5	7509	2429	2360	2360
q6	187	172	138	138
q7	948	766	629	629
q8	9504	1298	1056	1056
q9	7325	5398	5228	5228
q10	7021	2316	1893	1893
q11	500	286	278	278
q12	372	364	222	222
q13	18950	3714	3124	3124
q14	234	233	219	219
q15	646	490	480	480
q16	421	425	370	370
q17	639	853	396	396
q18	7493	7267	7204	7204
q19	1509	947	561	561
q20	362	384	238	238
q21	4172	3225	2426	2426
q22	1059	1042	1033	1033
Total cold run time: 119123 ms
Total hot run time: 34310 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5416	5130	5096	5096
q2	238	334	231	231
q3	2200	2691	2442	2442
q4	1503	1930	1469	1469
q5	4582	4454	4463	4454
q6	216	176	133	133
q7	1979	1938	1768	1768
q8	2599	2592	2514	2514
q9	7316	7146	7164	7146
q10	2976	3197	2778	2778
q11	568	501	489	489
q12	689	782	611	611
q13	3503	3947	3258	3258
q14	300	292	274	274
q15	531	481	463	463
q16	453	502	457	457
q17	1155	1526	1377	1377
q18	7727	7498	7576	7498
q19	774	769	858	769
q20	1939	2026	1884	1884
q21	4988	4383	4483	4383
q22	1067	1106	1024	1024
Total cold run time: 52719 ms
Total hot run time: 50518 ms

@doris-robot

doris-robot commented May 16, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193713 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46f8d99fd98eeffb86d3fd20ab0ac2d0d4a090fc, data reload: false
query1	1407	1086	1054	1054
query2	6024	1908	1916	1908
query3	10986	4549	4530	4530
query4	53298	24736	23088	23088
query5	5199	515	438	438
query6	357	213	200	200
query7	5031	515	298	298
query8	335	275	249	249
query9	6217	2698	2695	2695
query10	440	329	294	294
query11	15041	15065	14795	14795
query12	162	107	108	107
query13	1124	529	420	420
query14	10286	6492	6482	6482
query15	200	195	171	171
query16	6625	676	504	504
query17	1109	747	617	617
query18	1453	392	317	317
query19	212	202	186	186
query20	129	127	124	124
query21	213	126	107	107
query22	4496	4485	4344	4344
query23	34491	33587	33748	33587
query24	6755	2445	2469	2445
query25	484	485	422	422
query26	770	271	156	156
query27	2311	513	344	344
query28	3294	2183	2208	2183
query29	570	582	450	450
query30	277	220	190	190
query31	880	863	800	800
query32	73	68	61	61
query33	452	359	334	334
query34	814	892	515	515
query35	808	827	767	767
query36	943	1025	923	923
query37	110	97	76	76
query38	4234	4403	4321	4321
query39	1554	1477	1476	1476
query40	218	124	120	120
query41	54	54	53	53
query42	132	115	114	114
query43	534	530	510	510
query44	1342	841	850	841
query45	180	174	179	174
query46	861	1045	671	671
query47	1894	1869	1807	1807
query48	401	438	375	375
query49	710	510	445	445
query50	666	720	439	439
query51	4227	4360	4284	4284
query52	112	116	111	111
query53	233	263	189	189
query54	609	586	526	526
query55	87	97	88	88
query56	339	307	308	307
query57	1245	1228	1136	1136
query58	277	271	272	271
query59	2917	2999	2735	2735
query60	350	327	320	320
query61	128	122	129	122
query62	714	743	650	650
query63	232	187	189	187
query64	2227	1033	706	706
query65	4283	4205	4207	4205
query66	775	403	307	307
query67	16064	15734	15597	15597
query68	7860	874	517	517
query69	536	317	270	270
query70	1184	1095	1107	1095
query71	516	326	307	307
query72	5475	4747	4944	4747
query73	1226	639	359	359
query74	9070	9196	8881	8881
query75	3822	3232	2695	2695
query76	4242	1201	755	755
query77	621	361	307	307
query78	10183	10176	9317	9317
query79	2518	787	608	608
query80	625	498	456	456
query81	472	262	224	224
query82	430	124	98	98
query83	365	254	241	241
query84	290	119	82	82
query85	790	366	318	318
query86	391	320	301	301
query87	4461	4412	4313	4313
query88	3712	2306	2316	2306
query89	396	312	287	287
query90	1792	215	204	204
query91	143	146	115	115
query92	72	59	59	59
query93	2050	948	572	572
query94	656	407	303	303
query95	373	299	290	290
query96	507	574	284	284
query97	2762	2751	2620	2620
query98	232	211	204	204
query99	1442	1448	1310	1310
Total cold run time: 300189 ms
Total hot run time: 193713 ms

@doris-robot

doris-robot commented May 16, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46f8d99fd98eeffb86d3fd20ab0ac2d0d4a090fc, data reload: false
query1	0.04	0.03	0.03
query2	0.12	0.10	0.12
query3	0.24	0.20	0.19
query4	1.59	0.20	0.19
query5	0.46	0.44	0.44
query6	1.15	0.67	0.65
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.59	0.51	0.53
query10	0.57	0.58	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.59	0.61
query14	0.81	0.81	0.81
query15	0.88	0.88	0.85
query16	0.39	0.38	0.38
query17	1.07	1.07	1.04
query18	0.24	0.21	0.22
query19	1.96	1.77	1.82
query20	0.02	0.01	0.01
query21	15.40	0.88	0.56
query22	0.76	1.15	0.73
query23	14.89	1.37	0.62
query24	7.39	1.53	1.42
query25	0.55	0.30	0.09
query26	0.48	0.16	0.13
query27	0.05	0.05	0.05
query28	10.57	0.83	0.44
query29	12.57	3.95	3.27
query30	0.27	0.09	0.07
query31	2.81	0.59	0.38
query32	3.23	0.55	0.48
query33	3.00	3.04	3.04
query34	15.68	5.11	4.56
query35	4.48	4.49	4.46
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.15	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 104.43 s
Total hot run time: 29.74 s

@zddr

zddr commented May 19, 2025

Copy link
Copy Markdown
ContributorAuthor

run external

@zddr

zddr commented May 20, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented May 20, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 35251 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94ae9c96a763c05ce1ae272497c588bad8813ad8, data reload: false
------ Round 1 ----------------------------------
q1	26746	5279	5246	5246
q2	2085	291	184	184
q3	10396	1353	713	713
q4	10253	1068	570	570
q5	7911	2453	2499	2453
q6	205	176	132	132
q7	967	769	609	609
q8	9327	1527	1239	1239
q9	6858	5263	5299	5263
q10	6871	2348	1880	1880
q11	513	284	274	274
q12	348	378	219	219
q13	17784	3808	3093	3093
q14	256	247	213	213
q15	545	506	487	487
q16	428	437	373	373
q17	610	921	374	374
q18	7741	7274	7135	7135
q19	1206	1090	613	613
q20	355	381	227	227
q21	4107	3312	2997	2997
q22	1080	1027	957	957
Total cold run time: 116592 ms
Total hot run time: 35251 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5388	5366	5383	5366
q2	271	345	233	233
q3	2209	2685	2321	2321
q4	1581	2054	1527	1527
q5	4562	4380	4425	4380
q6	252	182	131	131
q7	2108	1928	1822	1822
q8	2845	2830	2748	2748
q9	7223	7134	7104	7104
q10	3043	3219	2753	2753
q11	621	512	494	494
q12	707	801	643	643
q13	3664	4021	3295	3295
q14	281	304	274	274
q15	537	482	479	479
q16	448	509	439	439
q17	1217	1686	1379	1379
q18	7554	7590	7491	7491
q19	875	863	908	863
q20	1935	1990	1889	1889
q21	4893	4395	4391	4391
q22	1054	1034	979	979
Total cold run time: 53268 ms
Total hot run time: 51001 ms

@doris-robot

doris-robot commented May 20, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 186865 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94ae9c96a763c05ce1ae272497c588bad8813ad8, data reload: false
query1	1028	491	506	491
query2	6566	1841	1832	1832
query3	6770	223	242	223
query4	26211	23313	23406	23313
query5	4337	623	463	463
query6	317	218	196	196
query7	4625	500	300	300
query8	304	261	245	245
query9	8645	2653	2661	2653
query10	491	345	282	282
query11	15816	15087	15086	15086
query12	166	114	110	110
query13	1669	560	420	420
query14	9335	6238	6206	6206
query15	203	192	179	179
query16	7345	626	457	457
query17	1169	699	559	559
query18	1980	409	327	327
query19	190	185	160	160
query20	129	123	121	121
query21	225	126	108	108
query22	4124	4119	4170	4119
query23	34024	33388	33196	33196
query24	8507	2419	2433	2419
query25	532	459	395	395
query26	1227	269	157	157
query27	2746	493	352	352
query28	4291	2107	2095	2095
query29	792	563	436	436
query30	284	210	190	190
query31	950	863	741	741
query32	73	67	62	62
query33	554	376	316	316
query34	805	851	539	539
query35	791	812	755	755
query36	937	994	893	893
query37	113	97	76	76
query38	4115	4118	4042	4042
query39	1463	1435	1413	1413
query40	212	120	110	110
query41	58	57	55	55
query42	129	116	107	107
query43	504	500	472	472
query44	1348	835	822	822
query45	177	178	169	169
query46	859	1021	665	665
query47	1777	1764	1689	1689
query48	400	455	319	319
query49	766	500	418	418
query50	637	686	437	437
query51	4140	4122	4116	4116
query52	101	100	99	99
query53	231	263	195	195
query54	592	591	515	515
query55	84	90	87	87
query56	319	309	279	279
query57	1174	1143	1082	1082
query58	270	253	259	253
query59	2552	2616	2517	2517
query60	333	319	313	313
query61	127	125	127	125
query62	801	705	666	666
query63	230	193	194	193
query64	4261	999	717	717
query65	4296	4248	4262	4248
query66	1149	425	330	330
query67	15862	15786	15448	15448
query68	7352	913	538	538
query69	491	330	286	286
query70	1139	1195	1115	1115
query71	403	347	315	315
query72	5684	4629	4644	4629
query73	629	595	359	359
query74	8967	9159	8844	8844
query75	3182	3216	2702	2702
query76	3248	1204	776	776
query77	528	404	381	381
query78	10212	10192	9269	9269
query79	2671	816	579	579
query80	635	518	446	446
query81	507	250	214	214
query82	708	129	94	94
query83	263	250	247	247
query84	243	109	81	81
query85	815	398	312	312
query86	382	305	265	265
query87	4548	4456	4415	4415
query88	4033	2324	2309	2309
query89	388	315	281	281
query90	1929	212	208	208
query91	142	147	112	112
query92	77	59	61	59
query93	2475	922	587	587
query94	727	419	314	314
query95	384	304	284	284
query96	493	567	283	283
query97	2726	2742	2688	2688
query98	238	213	206	206
query99	1326	1423	1294	1294
Total cold run time: 274783 ms
Total hot run time: 186865 ms

@doris-robot

doris-robot commented May 20, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94ae9c96a763c05ce1ae272497c588bad8813ad8, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.26	0.19	0.19
query4	1.60	0.20	0.19
query5	0.45	0.43	0.44
query6	1.55	0.64	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.53	0.52
query10	0.57	0.59	0.58
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.83	0.80	0.81
query15	0.88	0.86	0.87
query16	0.39	0.38	0.38
query17	1.07	1.03	1.01
query18	0.22	0.21	0.21
query19	1.96	1.81	1.85
query20	0.01	0.01	0.01
query21	15.40	0.90	0.58
query22	0.77	1.26	0.64
query23	14.85	1.35	0.61
query24	7.20	2.15	0.49
query25	0.50	0.20	0.06
query26	0.52	0.16	0.13
query27	0.05	0.05	0.05
query28	9.32	0.86	0.45
query29	12.57	3.96	3.35
query30	0.26	0.09	0.06
query31	2.83	0.59	0.38
query32	3.23	0.56	0.48
query33	3.08	3.04	3.11
query34	15.82	5.08	4.53
query35	4.52	4.50	4.46
query36	0.66	0.50	0.49
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.58 s
Total hot run time: 28.74 s

@github-actions

github-actionsBot commented May 20, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 20, 2025
@morrySnow
morrySnow merged commit 986f46b into apache:masterMay 20, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@zddrkoarz
[enhance](mtmv)cache table snapshot in refresh context (apache#50855)
f14ce0a
Assuming the SQL definition below, there are two tables (t1, t2) and one
materialized view (mv1).
When determining whether mv1 is synchronized with its base tables, the
system compares:
Whether mvp1 is synchronized with partition p1 and base table t2
Whether mvp2 is synchronized with partition p2 and base table t2
Optimization in this PR:
The original logic would fetch t2's snapshot information twice. This PR
improves efficiency by caching t2's snapshot in the refresh context to
avoid redundant retrieval.
CREATE TABLE t1
(
k2 TINYINT,
k3 INT not null
)
PARTITION BY LIST(`k3`)
(
PARTITION `p1` VALUES IN ('1'),
PARTITION `p2` VALUES IN ('2')
)
CREATE TABLE t2
(
k2 TINYINT,
k3 INT not null
);
create materialized view mv1 partition by(k3)
as
select * from t1 join t2;
mv1 will has two partition mvp1,mvp2
zddr added a commit to zddr/incubator-doris that referenced this pull request Jun 4, 2025
@zddr
[enhance](mtmv)cache table snapshot in refresh context (apache#50855)
450d2c7
Assuming the SQL definition below, there are two tables (t1, t2) and one
materialized view (mv1).
When determining whether mv1 is synchronized with its base tables, the
system compares:
Whether mvp1 is synchronized with partition p1 and base table t2
Whether mvp2 is synchronized with partition p2 and base table t2
Optimization in this PR:
The original logic would fetch t2's snapshot information twice. This PR
improves efficiency by caching t2's snapshot in the refresh context to
avoid redundant retrieval.
CREATE TABLE t1
(
k2 TINYINT,
k3 INT not null
)
PARTITION BY LIST(`k3`)
(
PARTITION `p1` VALUES IN ('1'),
PARTITION `p2` VALUES IN ('2')
)
CREATE TABLE t2
(
k2 TINYINT,
k3 INT not null
);
create materialized view mv1
partition by(k3)
as
select * from t1 join t2;
mv1 will has two partition mvp1,mvp2
@zddrzddr mentioned this pull request Jun 4, 2025
Merged
zddr added a commit to zddr/incubator-doris that referenced this pull request Jun 4, 2025
@zddr
[enhance](mtmv)cache table snapshot in refresh context (apache#50855)
5ddb936
Assuming the SQL definition below, there are two tables (t1, t2) and one
materialized view (mv1).
When determining whether mv1 is synchronized with its base tables, the
system compares:
Whether mvp1 is synchronized with partition p1 and base table t2
Whether mvp2 is synchronized with partition p2 and base table t2
Optimization in this PR:
The original logic would fetch t2's snapshot information twice. This PR
improves efficiency by caching t2's snapshot in the refresh context to
avoid redundant retrieval.
CREATE TABLE t1
(
k2 TINYINT,
k3 INT not null
)
PARTITION BY LIST(`k3`)
(
PARTITION `p1` VALUES IN ('1'),
PARTITION `p2` VALUES IN ('2')
)
CREATE TABLE t2
(
k2 TINYINT,
k3 INT not null
);
create materialized view mv1
partition by(k3)
as
select * from t1 join t2;
mv1 will has two partition mvp1,mvp2
@zddrzddr mentioned this pull request Jun 4, 2025
Merged
yiguolei pushed a commit that referenced this pull request Jun 5, 2025
@zddr
branch-2.1:[enhance](mtmv)cache table snapshot in refresh context (#5…
831d2c1
…0855) (#51493)
pick: #50855
dataroaring pushed a commit that referenced this pull request Jun 12, 2025
@zddr
branch-3.0:[enhance](mtmv)cache table snapshot in refresh context (#5…
c971c73
…0855) (#51494)
pick: #50855
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnowmorrySnowmorrySnow approved these changes
@seawindeseawindeseawinde approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.11-mergeddev/3.0.7-mergeddev/3.1.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zddr@hello-stephen@doris-robot@morrySnow@seawinde@yiguolei@dataroaring