[fix](schema-change) Fix sync MV loss due to index change by Hastyshell · Pull Request #54285 · apache/doris

Hastyshell · 2025-08-04T09:01:41Z

What problem does this PR solve?

Problem Summary:
Creation of shadow index did not take define stmt and where clause of MV into account, therefore, after index creation,
MV info will be lost.

This bug only affect index change on MV related columns. Other types of heavy sc are not allowed on MV related columns.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-08-04T09:01:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Hastyshell · 2025-08-04T09:01:59Z

run buildall

doris-robot · 2025-08-04T09:20:13Z

TPC-H: Total hot run time: 33877 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 928760273d2a5899c3707e1d6a60f811cd8e4344, data reload: false
------ Round 1 ----------------------------------
q1	17584	5408	5144	5144
q2	1946	273	184	184
q3	10320	1321	704	704
q4	10230	1002	535	535
q5	7537	2338	2320	2320
q6	181	171	139	139
q7	889	747	633	633
q8	9322	1304	966	966
q9	6848	5205	5149	5149
q10	6902	2361	1976	1976
q11	459	300	269	269
q12	359	373	238	238
q13	17823	3441	3055	3055
q14	228	248	220	220
q15	555	459	466	459
q16	430	443	374	374
q17	575	841	348	348
q18	7568	7165	6997	6997
q19	1808	987	530	530
q20	334	322	213	213
q21	3512	3167	2358	2358
q22	1099	1066	1077	1066
Total cold run time: 106509 ms
Total hot run time: 33877 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5384	5421	5429	5421
q2	246	312	221	221
q3	2115	2605	2207	2207
q4	1330	1732	1318	1318
q5	4167	4467	4521	4467
q6	216	178	147	147
q7	1893	1957	2068	1957
q8	2578	2428	2536	2428
q9	7205	7335	7289	7289
q10	3235	3294	2998	2998
q11	571	513	503	503
q12	667	817	608	608
q13	3426	3746	3252	3252
q14	285	305	285	285
q15	493	454	446	446
q16	454	489	461	461
q17	1176	1458	1388	1388
q18	8210	7794	7703	7703
q19	10247	910	1101	910
q20	1976	1996	1845	1845
q21	15029	4276	4269	4269
q22	1065	1071	994	994
Total cold run time: 71968 ms
Total hot run time: 51117 ms

doris-robot · 2025-08-04T09:32:11Z

TPC-DS: Total hot run time: 161789 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 928760273d2a5899c3707e1d6a60f811cd8e4344, data reload: false
reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL ============================================
query1	1028	378	421	378
query2	6516	1893	1695	1695
query3	6739	228	218	218
query4	26992	23400	23295	23295
query5	4375	668	517	517
query6	329	238	226	226
query7	4630	527	314	314
query8	286	245	225	225
query9	8628	2986	2969	2969
query10	487	362	307	307
query11	15785	15227	14748	14748
query12	187	145	132	132
query13	1667	559	428	428
query14	9368	5942	6039	5942
query15	212	201	184	184
query16	7137	692	517	517
query17	1195	789	660	660
query18	2105	495	336	336
query19	237	231	196	196
query20	154	154	138	138
query21	222	131	110	110
query22	4105	4010	3893	3893
query23	34658	34688	34478	34478
query24	7375	2425	2486	2425
query25	614	524	447	447
query26	1245	301	163	163
query27	2556	495	352	352
query28	4359	2353	2337	2337
query29	846	670	509	509
query30	311	243	211	211
query31	907	795	717	717
query32	94	81	87	81
query33	591	430	370	370
query34	858	881	541	541
query35	793	886	766	766
query36	1069	1050	936	936
query37	144	113	96	96
query38	3969	4026	3974	3974
query39	1454	1386	1379	1379
query40	248	151	136	136
query41	60	57	57	57
query42	143	125	136	125
query43	529	520	484	484
query44	1484	887	877	877
query45	210	196	186	186
query46	993	1113	697	697
query47	1789	1841	1743	1743
query48	412	445	336	336
query49	763	546	434	434
query50	709	746	446	446
query51	4170	4246	4143	4143
query52	129	133	128	128
query53	273	309	216	216
query54	667	675	562	562
query55	94	91	95	91
query56	379	375	356	356
query57	1192	1212	1120	1120
query58	350	340	349	340
query59	2676	2737	2606	2606
query60	411	403	419	403
query61	124	125	122	122
query62	849	742	688	688
query63	263	227	217	217
query64	4431	1090	769	769
query65	4235	4162	4188	4162
query66	1143	447	333	333
query67	query68	18108	900	999	900
query69	1481	279	290	279
query70	1371	1125	1140	1125
query71	676	324	326	324
query72	9126	2342	2114	2114
query73	3759	721	363	363
query74	9058	8812	8653	8653
query75	7463	3153	2651	2651
query76	8634	1260	805	805
query77	1161	428	343	343
query78	9682	10940	query79	15593	683	591	591
query80	3342	582	503	503
query81	544	263	234	234
query82	531	150	126	126
query83	379	297	275	275
query84	304	100	94	94
query85	1025	371	338	338
query86	365	335	299	299
query87	4225	4203	4149	4149
query88	5680	2261	2242	2242
query89	508	371	323	323
query90	2491	240	242	240
query91	148	143	112	112
query92	89	74	70	70
query93	6659	1000	672	672
query94	1402	395	292	292
query95	431	339	323	323
query96	527	625	286	286
query97	2739	2738	2574	2574
query98	253	236	225	225
query99	1466	1434	1277	1277
Total cold run time: 302491 ms
Total hot run time: 161789 ms

doris-robot · 2025-08-04T09:37:40Z

ClickBench: Total hot run time: 32.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 928760273d2a5899c3707e1d6a60f811cd8e4344, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.60	0.11	0.11
query5	0.43	0.42	0.43
query6	1.15	0.65	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.46	0.45
query10	0.53	0.52	0.51
query11	0.15	0.10	0.11
query12	0.15	0.11	0.11
query13	0.65	0.65	0.66
query14	0.90	1.27	0.92
query15	0.91	0.88	0.87
query16	0.38	0.40	0.41
query17	1.07	1.05	1.02
query18	0.22	0.21	0.20
query19	2.08	1.77	1.96
query20	0.01	0.02	0.01
query21	15.39	0.86	0.54
query22	0.77	1.17	0.65
query23	14.97	1.17	0.64
query24	6.81	1.60	0.81
query25	0.53	0.36	0.06
query26	0.42	0.16	0.14
query27	0.07	0.06	0.06
query28	9.93	0.84	0.45
query29	12.76	3.84	3.33
query30	3.04	2.99	2.95
query31	2.82	0.57	0.40
query32	3.24	0.58	0.50
query33	3.04	3.17	3.23
query34	16.15	5.33	4.93
query35	4.94	4.97	5.02
query36	0.69	0.52	0.50
query37	0.10	0.07	0.07
query38	0.06	0.05	0.04
query39	0.03	0.03	0.03
query40	0.17	0.13	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.05	0.04	0.03
Total cold run time: 107.33 s
Total hot run time: 32.83 s

Hastyshell · 2025-08-04T12:08:22Z

run buildall

hello-stephen · 2025-08-04T13:18:07Z

FE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-08-04T13:46:52Z

TPC-H: Total hot run time: 33500 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd11f8275a13140ccfda9b18dd65432f72d25a50, data reload: false
------ Round 1 ----------------------------------
q1	17679	5199	5280	5199
q2	1934	267	184	184
q3	10698	1288	720	720
q4	10354	1012	507	507
q5	10116	2322	2351	2322
q6	235	167	131	131
q7	901	753	634	634
q8	9309	1310	1066	1066
q9	7308	5072	5031	5031
q10	6967	2371	1981	1981
q11	487	279	267	267
q12	359	390	212	212
q13	17796	3469	2959	2959
q14	226	231	226	226
q15	545	457	464	457
q16	422	430	371	371
q17	572	837	342	342
q18	7222	7026	6856	6856
q19	1443	972	530	530
q20	312	323	213	213
q21	3447	3024	2293	2293
q22	1043	1085	999	999
Total cold run time: 109375 ms
Total hot run time: 33500 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5443	5301	5337	5301
q2	280	308	205	205
q3	2076	2559	2181	2181
q4	1334	1716	1330	1330
q5	4359	4475	4458	4458
q6	228	174	132	132
q7	1976	1935	1746	1746
q8	2441	2462	2523	2462
q9	7352	7465	7042	7042
q10	3141	3361	2925	2925
q11	536	507	475	475
q12	765	751	610	610
q13	3498	3669	3218	3218
q14	303	304	278	278
q15	505	457	450	450
q16	438	500	521	500
q17	1167	1439	1363	1363
q18	7932	7727	7986	7727
q19	10958	866	867	866
q20	1861	1933	1722	1722
q21	10662	4203	4295	4203
q22	1042	1013	1004	1004
Total cold run time: 68297 ms
Total hot run time: 50198 ms

doris-robot · 2025-08-04T13:58:54Z

TPC-DS: Total hot run time: 170320 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd11f8275a13140ccfda9b18dd65432f72d25a50, data reload: false
============================================
query1	1036	402	388	388
query2	6524	1939	1699	1699
query3	6749	228	218	218
query4	26433	23557	22896	22896
query5	4264	609	551	551
query6	329	238	221	221
query7	4636	508	295	295
query8	281	240	235	235
query9	8569	2877	2889	2877
query10	463	338	296	296
query11	15939	15010	14779	14779
query12	192	135	134	134
query13	1660	560	417	417
query14	8580	5889	5876	5876
query15	208	183	172	172
query16	7762	662	485	485
query17	1607	791	646	646
query18	2053	465	336	336
query19	294	215	232	215
query20	154	152	141	141
query21	217	125	106	106
query22	4069	4052	3965	3965
query23	34552	34283	34185	34185
query24	8332	2357	2462	2357
query25	576	517	432	432
query26	1122	296	158	158
query27	2629	497	364	364
query28	4258	2285	2293	2285
query29	756	600	480	480
query30	294	246	205	205
query31	887	817	715	715
query32	85	75	76	75
query33	585	412	363	363
query34	808	857	518	518
query35	800	857	778	778
query36	1019	1028	927	927
query37	139	112	95	95
query38	4069	3944	3953	3944
query39	1439	1404	1354	1354
query40	239	145	131	131
query41	61	61	53	53
query42	145	120	131	120
query43	496	525	476	476
query44	1377	870	861	861
query45	202	198	183	183
query46	953	1061	656	656
query47	1779	1821	1747	1747
query48	402	435	321	321
query49	712	506	417	417
query50	640	683	426	426
query51	4170	4179	4165	4165
query52	126	136	128	128
query53	268	293	209	209
query54	649	646	562	562
query55	95	90	88	88
query56	367	350	343	343
query57	1195	1221	1130	1130
query58	336	327	329	327
query59	2577	2678	2503	2503
query60	411	403	386	386
query61	137	126	154	126
query62	808	736	665	665
query63	256	210	216	210
query64	4280	1107	784	784
query65	4319	4155	4136	4136
query66	1081	468	333	333
query67	query68	19343	975	600	600
query69	1001	309	288	288
query70	1439	1172	1122	1122
query71	655	331	331	331
query72	9122	2322	4785	2322
query73	4039	755	353	353
query74	8916	8905	8701	8701
query75	7183	3088	2603	2603
query76	8199	1204	786	786
query77	1212	402	321	321
query78	9690	11444	10237	10237
query79	14930	581	584	581
query80	2382	543	487	487
query81	591	269	228	228
query82	574	166	121	121
query83	438	281	258	258
query84	303	95	78	78
query85	1045	437	333	333
query86	375	314	302	302
query87	4176	4249	4075	4075
query88	5534	2208	2222	2208
query89	485	359	324	324
query90	2630	226	226	226
query91	145	137	111	111
query92	92	72	65	65
query93	6513	956	651	651
query94	1102	384	272	272
query95	409	332	313	313
query96	491	589	283	283
query97	2706	2696	2584	2584
query98	245	227	291	227
query99	1469	1365	1260	1260
Total cold run time: 309884 ms
Total hot run time: 170320 ms

doris-robot · 2025-08-04T14:04:23Z

ClickBench: Total hot run time: 33.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd11f8275a13140ccfda9b18dd65432f72d25a50, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.25	0.07	0.08
query4	1.62	0.10	0.11
query5	0.42	0.41	0.42
query6	1.17	0.65	0.66
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.56	0.47	0.46
query10	0.55	0.53	0.53
query11	0.16	0.10	0.10
query12	0.15	0.11	0.10
query13	0.65	0.66	0.63
query14	0.93	1.17	1.12
query15	0.89	0.88	0.87
query16	0.40	0.39	0.39
query17	1.04	1.08	1.07
query18	0.21	0.20	0.20
query19	1.98	1.79	1.82
query20	0.01	0.01	0.01
query21	15.38	0.86	0.56
query22	0.77	1.24	0.64
query23	14.87	1.20	0.65
query24	6.82	1.69	0.85
query25	0.51	0.21	0.15
query26	0.61	0.17	0.14
query27	0.05	0.06	0.06
query28	9.90	0.82	0.44
query29	12.66	3.80	3.27
query30	3.02	3.02	2.91
query31	2.82	0.57	0.40
query32	3.25	0.58	0.49
query33	3.01	3.20	3.19
query34	16.09	5.30	4.87
query35	4.83	5.02	5.04
query36	0.68	0.52	0.51
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.09	0.02	0.02
query42	0.03	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 106.96 s
Total hot run time: 33.11 s

Hastyshell · 2025-08-05T03:13:24Z

run p0

Hastyshell · 2025-08-05T03:13:45Z

run vault_p0

github-actions · 2025-08-07T12:10:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-08-07T12:10:40Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Problem Summary: Creation of shadow index did not take define stmt and where clause of MV into account, therefore, after index creation, MV info will be lost. This bug only affect index change on MV related columns. Other types of heavy sc are not allowed on MV related columns.

Problem Summary: Creation of shadow index did not take define stmt and where clause of MV into account, therefore, after index creation, MV info will be lost. This bug only affect index change on MV related columns. Other types of heavy sc are not allowed on MV related columns.

…54285 (#54782) picked from #54285

…) (apache#5671)

Hastyshelland others added 2 commits August 4, 2025 12:01

[fix](schema-change) Fix sync MV loss due to index change
0073413

add test
9287602

dataroaring added the usercase Important user case type label label Aug 4, 2025

Update test_index_change_after_mv.groovy
bd11f82

dataroaring added dev/3.0.x dev/3.1.x dev/2.1.x labels Aug 7, 2025

gavinchou approved these changes Aug 7, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 7, 2025

github-actionsBot added the reviewed label Aug 7, 2025

luwei16 approved these changes Aug 8, 2025
View reviewed changes

dataroaring merged commit f467914 into apache:masterAug 10, 2025
30 of 32 checks passed

github-actionsBot mentioned this pull request Aug 10, 2025
branch-3.0: [fix](schema-change) Fix sync MV loss due to index change #54285 #54530
Closed

github-actionsBot mentioned this pull request Aug 10, 2025
branch-2.1: [fix](schema-change) Fix sync MV loss due to index change #54285 #54531
Closed

github-actionsBot mentioned this pull request Aug 10, 2025
branch-3.1: [fix](schema-change) Fix sync MV loss due to index change #54285 #54533
Closed

morrySnow added the dev/3.1.x-conflict label Aug 13, 2025

morrySnow mentioned this pull request Aug 19, 2025
branch-3.1: [fix](schema-change) Fix sync MV loss due to index change #54285 #54782
Merged

morrySnow pushed a commit that referenced this pull request Aug 19, 2025
branch-3.1: [fix](schema-change) Fix sync MV loss due to index change#…
11554bf
…54285 (#54782) picked from #54285

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Aug 19, 2025

Hastyshell added a commit to Hastyshell/doris that referenced this pull request Nov 12, 2025
[fix](schema-change) Fix sync MV loss due to index change (apache#54285)
e5e564a

yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Dec 30, 2025
[fix](schema-change) Fix sync MV loss due to index change (apache#54285…
678a9c6
…) (apache#5671)

Uh oh!

Conversation

Hastyshell commented Aug 4, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 4, 2025

Uh oh!

Hastyshell commented Aug 4, 2025

Uh oh!

doris-robot commented Aug 4, 2025

Uh oh!

doris-robot commented Aug 4, 2025

Uh oh!

doris-robot commented Aug 4, 2025

Uh oh!

Hastyshell commented Aug 4, 2025

Uh oh!

hello-stephen commented Aug 4, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Aug 4, 2025

Uh oh!

doris-robot commented Aug 4, 2025

Uh oh!

doris-robot commented Aug 4, 2025

Uh oh!

Hastyshell commented Aug 5, 2025

Uh oh!

Hastyshell commented Aug 5, 2025

Uh oh!

github-actionsBot commented Aug 7, 2025

Uh oh!

github-actionsBot commented Aug 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Hastyshell commented Aug 4, 2025•
edited
Loading