[fix](inverted index)Support rename column with inverted index by qidaye · Pull Request #38079 · apache/doris

qidaye · 2024-07-18T08:48:59Z

Proposed changes

Copy columns and properties to make them mutable.

close#38078

doris-robot · 2024-07-18T08:49:04Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

qidaye · 2024-07-18T08:49:41Z

Cases will be added in another PR.
#38132

qidaye · 2024-07-18T08:49:47Z

run buildall

morrySnow

add a test case

github-actions · 2024-07-18T08:54:13Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-18T08:54:16Z

PR approved by anyone and no changes requested.

doris-robot · 2024-07-18T09:30:41Z

TPC-H: Total hot run time: 41132 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f95cbb52053db64123f0d55de73a9bd3e7a28b9, data reload: false
------ Round 1 ----------------------------------
q1	17604	4510	4440	4440
q2	2007	199	193	193
q3	10421	1326	1176	1176
q4	10191	857	820	820
q5	7558	2789	2771	2771
q6	240	139	139	139
q7	1034	621	624	621
q8	9219	2276	2255	2255
q9	8961	6811	6759	6759
q10	8893	3843	3929	3843
q11	454	247	245	245
q12	443	228	223	223
q13	18784	2972	2987	2972
q14	278	245	246	245
q15	553	492	488	488
q16	511	387	381	381
q17	1010	600	764	600
q18	8040	7507	7381	7381
q19	6315	1533	1530	1530
q20	662	351	342	342
q21	5061	3409	4023	3409
q22	365	300	299	299
Total cold run time: 118604 ms
Total hot run time: 41132 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4645	4420	4510	4420
q2	392	289	275	275
q3	3257	3053	2967	2967
q4	2102	1735	1688	1688
q5	5674	5571	5669	5571
q6	254	142	137	137
q7	2294	1810	1873	1810
q8	3478	3664	3635	3635
q9	9326	9485	9366	9366
q10	4085	4008	4021	4008
q11	626	506	494	494
q12	820	668	651	651
q13	16347	3186	3168	3168
q14	330	277	295	277
q15	555	496	493	493
q16	506	441	428	428
q17	1969	1612	1584	1584
q18	8239	7978	7966	7966
q19	1877	1818	1699	1699
q20	2246	1888	1872	1872
q21	5240	4859	5137	4859
q22	614	518	523	518
Total cold run time: 74876 ms
Total hot run time: 57886 ms

qidaye · 2024-07-18T09:33:06Z

run feut

doris-robot · 2024-07-18T09:41:52Z

TPC-DS: Total hot run time: 173515 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f95cbb52053db64123f0d55de73a9bd3e7a28b9, data reload: false
query1	913	381	369	369
query2	6370	1830	1859	1830
query3	6642	212	216	212
query4	27309	17533	17399	17399
query5	3776	459	473	459
query6	275	185	163	163
query7	4582	288	292	288
query8	244	213	192	192
query9	8640	2507	2494	2494
query10	432	279	261	261
query11	10507	10069	10049	10049
query12	113	81	81	81
query13	1627	379	357	357
query14	10259	7915	7491	7491
query15	210	164	165	164
query16	7618	312	305	305
query17	1391	543	530	530
query18	1835	281	269	269
query19	187	149	150	149
query20	90	82	82	82
query21	208	123	121	121
query22	4246	4005	3974	3974
query23	33744	33651	33513	33513
query24	11159	2940	2893	2893
query25	588	379	413	379
query26	704	147	145	145
query27	2322	279	280	279
query28	6251	2114	2118	2114
query29	874	664	616	616
query30	282	148	151	148
query31	1003	794	749	749
query32	94	53	55	53
query33	654	333	300	300
query34	943	497	502	497
query35	707	601	591	591
query36	1166	977	925	925
query37	149	85	80	80
query38	2939	2833	2799	2799
query39	871	826	821	821
query40	197	117	117	117
query41	46	42	45	42
query42	119	95	97	95
query43	510	471	468	468
query44	1085	724	727	724
query45	198	158	160	158
query46	1088	717	722	717
query47	1869	1760	1769	1760
query48	363	293	290	290
query49	861	401	431	401
query50	783	395	388	388
query51	6893	6843	6962	6843
query52	101	93	95	93
query53	375	292	283	283
query54	868	499	449	449
query55	71	74	72	72
query56	282	260	271	260
query57	1121	1068	1060	1060
query58	246	260	247	247
query59	2967	2632	2650	2632
query60	291	274	267	267
query61	95	93	88	88
query62	803	637	637	637
query63	313	285	286	285
query64	9159	2198	7514	2198
query65	3168	3096	3116	3096
query66	768	335	391	335
query67	15413	14918	14757	14757
query68	4515	534	560	534
query69	534	423	349	349
query70	1161	1139	1076	1076
query71	426	275	276	275
query72	7550	5622	5525	5525
query73	734	337	325	325
query74	6146	5663	5621	5621
query75	3382	2638	2625	2625
query76	2210	943	939	939
query77	463	288	294	288
query78	9537	9269	8904	8904
query79	3008	511	525	511
query80	1996	493	467	467
query81	582	214	227	214
query82	861	135	134	134
query83	290	164	164	164
query84	276	91	83	83
query85	1935	304	284	284
query86	475	306	301	301
query87	3274	3116	3054	3054
query88	4433	2456	2465	2456
query89	493	385	383	383
query90	1814	197	190	190
query91	133	101	100	100
query92	58	48	46	46
query93	5141	516	516	516
query94	1129	218	210	210
query95	402	330	316	316
query96	618	287	277	277
query97	3219	3021	2991	2991
query98	226	201	189	189
query99	1564	1270	1265	1265
Total cold run time: 280562 ms
Total hot run time: 173515 ms

doris-robot · 2024-07-18T09:47:25Z

ClickBench: Total hot run time: 30.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f95cbb52053db64123f0d55de73a9bd3e7a28b9, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.48	0.48
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.55	0.55
query11	0.15	0.12	0.12
query12	0.14	0.12	0.12
query13	0.58	0.59	0.58
query14	0.76	0.77	0.79
query15	0.84	0.85	0.81
query16	0.38	0.35	0.37
query17	0.99	0.97	1.06
query18	0.22	0.22	0.22
query19	1.86	1.76	1.80
query20	0.01	0.01	0.01
query21	15.39	0.75	0.66
query22	3.95	7.95	1.88
query23	18.29	1.31	1.28
query24	2.10	0.23	0.23
query25	0.15	0.08	0.09
query26	0.31	0.20	0.20
query27	0.45	0.24	0.24
query28	13.18	1.02	1.00
query29	12.61	3.38	3.36
query30	0.26	0.06	0.06
query31	2.87	0.41	0.39
query32	3.28	0.47	0.48
query33	2.91	2.93	2.90
query34	16.92	4.30	4.34
query35	4.39	4.43	4.49
query36	0.65	0.46	0.48
query37	0.20	0.16	0.15
query38	0.15	0.16	0.15
query39	0.05	0.03	0.03
query40	0.16	0.12	0.13
query41	0.08	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.2 s
Total hot run time: 30.72 s

qidaye · 2024-07-18T11:48:36Z

run external

qidaye · 2024-07-18T11:48:41Z

run feut

qidaye · 2024-07-18T11:48:45Z

run p0

qidaye · 2024-07-18T11:48:49Z

run p1

qidaye · 2024-07-18T13:34:00Z

run feut

qidaye · 2024-07-18T13:34:04Z

run p0

qidaye · 2024-07-18T13:34:07Z

run p1

Copy columns and properties to make them mutable.

qidaye · 2024-07-18T13:57:18Z

run buildall

doris-robot · 2024-07-18T14:33:29Z

TPC-H: Total hot run time: 39845 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f78db4630f754955c331d2e902c7538838bb93b, data reload: false
------ Round 1 ----------------------------------
q1	17635	4330	4319	4319
q2	2006	190	192	190
q3	10452	1136	1136	1136
q4	10182	811	763	763
q5	7602	2701	2669	2669
q6	221	140	138	138
q7	958	608	600	600
q8	9223	2060	2095	2060
q9	8793	6544	6533	6533
q10	8689	3857	3767	3767
q11	478	240	246	240
q12	391	226	229	226
q13	17873	2967	3013	2967
q14	263	228	238	228
q15	526	484	481	481
q16	486	384	371	371
q17	962	634	755	634
q18	8024	7437	7341	7341
q19	6259	1312	1425	1312
q20	668	343	330	330
q21	5017	3262	3261	3261
q22	350	281	279	279
Total cold run time: 117058 ms
Total hot run time: 39845 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4465	4321	4286	4286
q2	392	278	265	265
q3	3054	2918	2943	2918
q4	2052	1680	1753	1680
q5	5631	5598	5562	5562
q6	229	137	133	133
q7	2217	1918	1910	1910
q8	3273	3457	3398	3398
q9	8814	8885	8851	8851
q10	4157	3836	3879	3836
q11	625	508	501	501
q12	828	659	666	659
q13	16524	3376	3341	3341
q14	330	314	299	299
q15	539	477	499	477
q16	503	465	465	465
q17	1815	1597	1523	1523
q18	8420	8388	8083	8083
q19	3108	1698	1599	1599
q20	4083	1863	1926	1863
q21	9777	4940	4904	4904
q22	598	505	517	505
Total cold run time: 81434 ms
Total hot run time: 57058 ms

doris-robot · 2024-07-18T14:44:45Z

TPC-DS: Total hot run time: 173091 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f78db4630f754955c331d2e902c7538838bb93b, data reload: false
query1	916	368	369	368
query2	6470	1915	1863	1863
query3	6639	209	216	209
query4	28416	17574	17279	17279
query5	3652	482	463	463
query6	257	181	183	181
query7	4588	295	287	287
query8	244	198	206	198
query9	8466	2530	2500	2500
query10	456	291	280	280
query11	11717	10095	10131	10095
query12	118	88	84	84
query13	1648	378	372	372
query14	12417	6369	8090	6369
query15	262	164	165	164
query16	7420	325	321	321
query17	1804	552	528	528
query18	1234	278	272	272
query19	196	147	149	147
query20	88	84	79	79
query21	212	131	127	127
query22	4241	4057	4048	4048
query23	34389	33952	34053	33952
query24	10606	2910	2924	2910
query25	621	389	398	389
query26	1391	152	150	150
query27	2674	281	282	281
query28	7268	2108	2094	2094
query29	894	633	642	633
query30	255	157	159	157
query31	979	781	783	781
query32	97	52	55	52
query33	764	303	326	303
query34	1048	509	500	500
query35	724	577	606	577
query36	1146	967	1000	967
query37	154	86	90	86
query38	3007	2899	2829	2829
query39	902	864	801	801
query40	255	122	115	115
query41	47	43	44	43
query42	122	96	100	96
query43	506	460	480	460
query44	1246	753	724	724
query45	201	166	156	156
query46	1104	722	753	722
query47	1871	1775	1782	1775
query48	373	292	292	292
query49	859	406	416	406
query50	780	409	404	404
query51	6938	6828	6708	6708
query52	105	90	90	90
query53	362	289	294	289
query54	949	453	452	452
query55	77	72	73	72
query56	297	271	271	271
query57	1143	1070	1028	1028
query58	256	240	252	240
query59	2896	2824	2699	2699
query60	329	271	274	271
query61	98	93	97	93
query62	827	632	662	632
query63	318	284	287	284
query64	9800	2233	1711	1711
query65	3146	3096	3123	3096
query66	779	328	330	328
query67	15342	15041	14898	14898
query68	6059	562	560	560
query69	628	436	340	340
query70	1160	1089	1064	1064
query71	478	291	286	286
query72	7267	5849	5548	5548
query73	783	334	326	326
query74	6142	5747	5770	5747
query75	3669	2649	2686	2649
query76	3773	986	926	926
query77	634	319	306	306
query78	9457	9221	8998	8998
query79	4373	544	530	530
query80	1257	542	474	474
query81	586	223	219	219
query82	472	135	131	131
query83	197	166	167	166
query84	281	95	88	88
query85	980	316	296	296
query86	352	329	303	303
query87	3308	3128	3083	3083
query88	3611	2467	2471	2467
query89	497	392	378	378
query90	1892	191	190	190
query91	127	103	100	100
query92	61	49	48	48
query93	4702	530	519	519
query94	1170	223	214	214
query95	408	318	330	318
query96	616	275	279	275
query97	3193	3037	3052	3037
query98	221	208	194	194
query99	1543	1266	1260	1260
Total cold run time: 289303 ms
Total hot run time: 173091 ms

doris-robot · 2024-07-18T14:50:16Z

ClickBench: Total hot run time: 30.07 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f78db4630f754955c331d2e902c7538838bb93b, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.08
query5	0.51	0.48	0.49
query6	1.15	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.48
query10	0.54	0.53	0.54
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.59
query14	0.75	0.77	0.78
query15	0.85	0.83	0.83
query16	0.35	0.36	0.37
query17	0.96	0.95	1.02
query18	0.23	0.22	0.22
query19	1.83	1.70	1.67
query20	0.02	0.01	0.01
query21	15.39	0.73	0.65
query22	4.66	7.40	1.55
query23	18.33	1.33	1.24
query24	2.15	0.24	0.22
query25	0.17	0.10	0.09
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.31	1.00	1.00
query29	12.58	3.32	3.27
query30	0.25	0.06	0.05
query31	2.86	0.39	0.39
query32	3.33	0.48	0.46
query33	2.89	2.89	2.94
query34	16.95	4.31	4.35
query35	4.37	4.38	4.53
query36	0.66	0.48	0.47
query37	0.19	0.15	0.17
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.06
query43	0.04	0.04	0.03
Total cold run time: 110.08 s
Total hot run time: 30.07 s

github-actions · 2024-07-19T06:06:23Z

PR approved by at least one committer and no changes requested.

zzzxl1993

LGTM

eldenmoon

LGTM

…e#38079) Copy columns and properties to make them mutable. closeapache#38078

… (#38121) ## Proposed changes backport #38079

Copy columns and properties to make them mutable. close#38078

## Proposed changes 1. Add exception when trying to change index storage format 2. Add test cases for index format v2 - test_add_build_index_with_format_v2 - test_create_table_with_format_v2 - test_cumulative_compaction_with_format_v2 - test_drop_column_with_format_v2 - test_drop_index_with_format_v2 - test_index_change_format - test_mor_table_with_format_v2 - test_mow_table_with_format_v2 - test_recover_with_format_v2 - test_rename_column_with_format_v2 - test_single_replica_compaction_with_format_v2 3. There is a test case `test_rename_column_with_format_v2` for #38079

…38132) ## Proposed changes 1. Add exception when trying to change index storage format 2. Add test cases for index format v2 - test_add_build_index_with_format_v2 - test_create_table_with_format_v2 - test_cumulative_compaction_with_format_v2 - test_drop_column_with_format_v2 - test_drop_index_with_format_v2 - test_index_change_format - test_mor_table_with_format_v2 - test_mow_table_with_format_v2 - test_recover_with_format_v2 - test_rename_column_with_format_v2 - test_single_replica_compaction_with_format_v2 3. There is a test case `test_rename_column_with_format_v2` for apache#38079

## Proposed changes 1. Add exception when trying to change index storage format 2. Add test cases for index format v2 - test_add_build_index_with_format_v2 - test_create_table_with_format_v2 - test_cumulative_compaction_with_format_v2 - test_drop_column_with_format_v2 - test_drop_index_with_format_v2 - test_index_change_format - test_mor_table_with_format_v2 - test_mow_table_with_format_v2 - test_recover_with_format_v2 - test_rename_column_with_format_v2 - test_single_replica_compaction_with_format_v2 3. There is a test case `test_rename_column_with_format_v2` for #38079

…38132) ## Proposed changes 1. Add exception when trying to change index storage format 2. Add test cases for index format v2 - test_add_build_index_with_format_v2 - test_create_table_with_format_v2 - test_cumulative_compaction_with_format_v2 - test_drop_column_with_format_v2 - test_drop_index_with_format_v2 - test_index_change_format - test_mor_table_with_format_v2 - test_mow_table_with_format_v2 - test_recover_with_format_v2 - test_rename_column_with_format_v2 - test_single_replica_compaction_with_format_v2 3. There is a test case `test_rename_column_with_format_v2` for apache#38079

…e#38079) Copy columns and properties to make them mutable. closeapache#38078

…38132) ## Proposed changes 1. Add exception when trying to change index storage format 2. Add test cases for index format v2 - test_add_build_index_with_format_v2 - test_create_table_with_format_v2 - test_cumulative_compaction_with_format_v2 - test_drop_column_with_format_v2 - test_drop_index_with_format_v2 - test_index_change_format - test_mor_table_with_format_v2 - test_mow_table_with_format_v2 - test_recover_with_format_v2 - test_rename_column_with_format_v2 - test_single_replica_compaction_with_format_v2 3. There is a test case `test_rename_column_with_format_v2` for apache#38079

morrySnow reviewed Jul 18, 2024
View reviewed changes

morrySnow previously approved these changes Jul 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2024

github-actionsBot added the reviewed label Jul 18, 2024

starocean999 previously approved these changes Jul 18, 2024
View reviewed changes

[fix](inverted index)Support rename column with inverted index
6f78db4
Copy columns and properties to make them mutable.

qidaye dismissed stale reviews from starocean999 and morrySnow via 6f78db4July 18, 2024 13:56

qidaye force-pushed the fix_rename_column_with_index branch from 0f95cbb to 6f78db4Compare July 18, 2024 13:56

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jul 18, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 19, 2024

zzzxl1993 approved these changes Jul 19, 2024
View reviewed changes

eldenmoon approved these changes Jul 19, 2024
View reviewed changes

qidaye merged commit 2c4aee4 into apache:masterJul 19, 2024

qidaye deleted the fix_rename_column_with_index branch July 19, 2024 06:29

qidaye added the dev/2.1.x label Jul 19, 2024

qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jul 19, 2024
[fix](inverted index)Support rename column with inverted index (apach…
366f443
…e#38079) Copy columns and properties to make them mutable. closeapache#38078

This was referenced Jul 19, 2024
[fix](inverted index)Support rename column with inverted index (#38079) #38121
Merged
[test](inverted index)Add cases for inverted index format v2 #38132
Merged

airborne12 pushed a commit that referenced this pull request Jul 19, 2024
[fix](inverted index)Support rename column with inverted index (#38079)…
dfc82a4
… (#38121) ## Proposed changes backport #38079

wm1581066 added dev/2.1.5-merged and removed dev/2.1.x labels Jul 19, 2024

dataroaring pushed a commit that referenced this pull request Jul 22, 2024
[fix](inverted index)Support rename column with inverted index (#38079)
8aac782
Copy columns and properties to make them mutable. close#38078

xiaokang added 2.1.0-conflict dev/2.0.x dev/3.0.x doing labels Jul 24, 2024

dataroaring added dev/3.0.1-merged and removed dev/3.0.x labels Jul 25, 2024

qidaye removed the 2.1.0-conflict label Aug 1, 2024

gavinchou mentioned this pull request Aug 19, 2024
Release Note 3.0.1 #39570
Closed

airborne12 removed dev/2.0.x doing labels Nov 7, 2024

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](inverted index)Support rename column with inverted index (apach…
8100b31
…e#38079) Copy columns and properties to make them mutable. closeapache#38078

Uh oh!

Conversation

qidaye commented Jul 18, 2024

Proposed changes

Uh oh!

doris-robot commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

morrySnow left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 18, 2024

Uh oh!

github-actionsBot commented Jul 18, 2024

Uh oh!

doris-robot commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

doris-robot commented Jul 18, 2024

Uh oh!

doris-robot commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

qidaye commented Jul 18, 2024

Uh oh!

doris-robot commented Jul 18, 2024

Uh oh!

doris-robot commented Jul 18, 2024

Uh oh!

doris-robot commented Jul 18, 2024

Uh oh!

github-actionsBot commented Jul 19, 2024

Uh oh!

zzzxl1993 left a comment

Choose a reason for hiding this comment

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

qidaye commented Jul 18, 2024•
edited
Loading